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ABSTRACT 


As it stands today, software is the major expense in software-intensive weapons 
systems. Therefore software is, and should be treated as a eapital investment and an 
approaeh emphasizing a strategie investment methodology in its aequisition is neeessary. 
The strategie flexibility in software engineering deeisions can be valued as a portfolio of 
options or real assets, much akin to options on financial securities which have real 
economic value under uncertainty. This approach would emphasize the linking of 
program management decisions to current and future unknown situations within the 
stipulated parameters of cost, schedule and functionality thus giving the managers a set of 
choices or options. 

This dissertation describes a strategic decision-making process that is based on 
the general concepts of initiating the software acquisition process with a situation 
assessments phase, identifying un-resolvable high-level uncertainties, generating the 
appropriate strategic actions and deriving the benefits or value created either explicitly or 
in the form of Real Options. We present a framework based on Real Options theory to 
allow decision makers to better balance customer requirements as dictated by operational 
needs within financial viability and schedule constraints through the identification, 
valuation and optimization of strategic decision pathways created in the form of Real 
Options. 

We apply the framework to the software component (Future Combat Systems 
Network) of U.S. Army Future Combat System (FCS). Our study found that when 
properly formulated, a Real Options approach could be used as an effective risk 
management tool to guide decision-making at the software acquisition level further 
complementing the risk-driven spiral development approach currently being utilized in 
the U.S. Department of Defense (DoD) evolutionary acquisition model. 
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1. INTRODUCTION 


A. MOTIVATION 

Software engineering is defined as an engineering diseipline eoneerned with all 
aspeets of software produetion. Given this definition, software in itself is a eomplex 
intangible artifaet, acquired and produced via a set of decision-making activities 
integrated with a software engineering process with the goal being to deliver a product 
that meets the customers’ needs and requirements within the stipulated cost and schedule 
constraints. The role of software as a “technology platform” in U.S. Department of 
Defense (DoD) weapons systems has steadily increased over the last 40 years (Figure 1) 
jumping from providing a mere 8% of weapons systems functionality in 1960 to 
providing 80% of weapons systems functionality in 2000. 
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Figure 1. Software Growth in Weapons Systems (From: [10]). 


Thus, considering the immense presence and ever-increasing role which software 
plays not only in weapons systems, but in our every day lives, industry and government, 
software can be viewed as an inexhaustible technological resource, vital and necessary 
for the good functioning and evolution of modern day society [4]. 
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The software aequisition lifeeycle, which encapsulates the activities related to its 
procurement, development, implementation and subsequent maintenance, continues to 
present challenges to software executives and program managers due to increasingly 
complex organizational requirements and the ever increasing role which software plays in 
U.S. Department of Defense (DoD) weapons systems. Various factors and considerations, 
most of which are complex in nature compound the software acquisition process, factors 
which present themselves in the form of “uncertainties”, and which have the potential of 
introducing risks if the uncertainties are not adequately addressed and or resolved. 
Consequently, an integrated decision-making approach that explicitly addresses the risks 
associated with the pertinent software acquisition activities of procurement methods, 
software development approaches, implementation strategy and future evolution of the 
software needs to be considered to better guide the software acquisition decision-making 
process. 

B. STATEMENT OF THE PROBLEM 

In the U.S. DoD, technology acquisitions in the form of software intensive 
weapons systems serves as the cornerstone of the transformation strategy currently 
adopted by the U.S. Military in its efforts to modernize its fleet of weapons systems for 
future conflicts. However, the benefits of these force “enablers” continue to be plagued 
by massive cost and schedule overruns. The resulting impact has often led to a reduction 
in the scope of desired functionality as depicted in Table 1 below which leaves the war¬ 
fighters needs unfulfilled. 


Program 

Initial 

Investment 

Initial 

Quantity 

Latest 

Investment 

Latest 

Quantity 

% Unit 
Cost 
Increase 

% 

Quantity 

Decrease 

Joint Strike 
Fighter 

$189.8 billion 

2,866 

aireraft 

$206.3 billion 

2,459 

aireraft 

26.7 

14.2 

Future Combat 
Systems 

$92 billion 

18 System 

$163.7 billion 

14 

systems 

54.4 

22.3 

F-22A Raptor 

$81.1 billion 

648 aireraft 

$65.4 billion 

181 

aireraft 

188.7 

72.1 


Table 1. Program Management Failures of Top Three Major Weapons Systems. ^ 


1 Numbers were eomplied from various GAO reports and were eurrent as of 2007. 
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This dilemma is highlighted in a 2007 interview of the Army’s Program 
Exeeutive Offieer (PEO) for Ammunition, in whieh “the ability to aequire and maintain, 
safe, reliable supportable and modifiable software systems whieh met user requirements 
in an environment of rapid teehnologieal advanees” was identified as their biggest 
ehallenge in software aequisitions [5]. Eurthermore, the El.S. Government Aeeountability 
Offiee (GAO) responsible for reviewing weapon systems investments found eonsistent 
problems of eost inereases, schedule delays, and performance shortfalls exacerbated by 
factors such as pressure on program managers to promise more than they could deliver. 
These concerns infer a resounding theme which continues to be resonated within the 
software acquisition community: Meeting customer requirements within cost and 
schedule constraints. This has led to calls for reform in the DoD acquisition strategy and 
the incorporation of broad improvement strategies which embraces the best practices of 
software-intensive systems acquisition from the commercial sector. 

Balancing the satisfaction of a customer’s ever-changing requirements within the 
realms of meeting both current and future uncertain operational needs against the costs 
and schedule constraints poses a cumbersome challenge to the software executive, 
thereby making software-investments a very risky venture; risky in the sense that 
software engineering and investment decisions are plagued by uncertainties which more 
often than not leads to varying degrees of risk ranging from operational shortfalls to cost 
and schedule overruns. 

The inefficiencies of current management techniques as shown in the acquisition 
failures (Table 1) (failure to meet war-fighter’s needs on time and schedule) highlight the 
needs of new management approaches that proactively plan for, and factor in uncertainty 
into their acquisition strategy. This is because the acquisition of software, its 
development and the operational use of the software are all dominated by human action, 
human judgment and decision making, and inevitably human error [2]. The outcome is, 
therefore, often uncertain and unpredictable, and leads to unavoidable uncertainties [2]. 
Thus the highly uncertain nature of software investment decisions makes the 
management of uncertainties a major issue in software engineering. The work of Ziv and 
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Richardson, which led to the proposal of the “Uncertainty Principle in Software 
Engineering ” (UPSE), laid out the groundwork for identifying the software development 
uneertainties faeing the software projeet manager and the Real Options approaeh 
proposed in this study is an attempt to identify, quantify, value, hedge and manage the 
uneertainties whieh introduees both teehnieal and managerial risks in the form of produet 
quality and eost and sehedule overrun. 

C. RESEARCH OBJECTIVES 

This researeh takes a holistie approaeh towards addressing both the teehnieal and 
managerial risks whieh plague software aequisitions through the applieation and 
employment of teehniques based on finaneial options theory (Real Options). As it stands 
today, software is the major expense in software intensive systems. Therefore software is, 
and should be treated as a eapital investment and an approaeh emphasizing a strategie 
investment methodology in its aequisition is neeessary. This approaeh would emphasize 
the linking of program management deeisions to eurrent and future unknown situations 
within the stipulated parameters of eost, sehedule and funetionality thus giving the 
managers a set of choices or options. 

These ehoiees are ealled “Real Options” and originate from researeh done to priee 
finaneial option eontraets in the field of finaneial derivatives. Its underlying premise is 
based on the reeognition that strategie flexibility in software engineering deeisions ean be 
valued as a portfolio of options or real assets, mueh akin to options on finaneial seeurities 
whieh have real eeonomie value under uneertainty [9]. It eenters on real (non-finaneial) 
assets and is valuable beeause they enable the option holder (software program manager) 
to take advantage of potential benefits while eontrolling risk. 

It is assumed, that when employed within the eontext of a software-related eapital 
investment effort, as a strategie deeision-making framework a Real Options approaeh 
makes the ease that strategie aetion identifies and ereates valuable options whieh eould be 
valued and exereised (if appropriate), starting the eyele of value ereation and new options 
all over again [6]. This strategie deeision-making proeess (Figure 2) is based on the 
general eoneepts of initiating the software aequisition proeess with a situation 
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assessments phase, generating the appropriate strategic actions and deriving the benefits 
or value created either explicitly or in the form of Real Options. While risks associated 
with large-scale software-related capital investments or acquisitions have been effectively 
managed through the application of stochastic frameworks, we believe that a framework 
based on Real Options theory would better balance customer requirements as dictated by 
operational needs within financial viability and schedule constraints through the 
identification, valuation and optimization of strategic decision pathways created in the 
form of Real Options. 


Assessment 


Acrion 


Result 



Figure 2. Strategy Formulation and Real Options (From: [6]) 


The Real Options approach calls for the existence or satisfaction of certain pre¬ 
conditions before it can be applied which we believe are directly correlated to the various 
activities associated with software related capital investments. These pre-conditions as 
outlined in [14] call for the following: 

1. The existence of a basic financial model used to evaluate the costs and 
benefits of the underlying software asset (e.g. Net Present Value (NPV) as 
the Real Options approach builds on the existing tried-and-tested 
approaches of current financial modeling techniques. 

2. The existence of uncertainties during the software-related capital 
investment decision-making process otherwise, the Real Options analysis 
becomes useless as everything is assumed to be certain and known. 

3. The uncertainties surrounding the software-related capital investment 
decision-making process must introduce risks which directly impact the 
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decision-making process. Real Options could then be used to hedge the 
downside risk and take advantage of the upside uncertainties. 

4. Management must have the flexibility or option to make mid course 
corrections when actively managing the project. 

5. Management must be smart enough to execute the Real Options when it 
becomes optimal to do so. 

This study seeks to investigate the application of the Real Options approach to 
both the technical and managerial risks associated with software-related capital 
investments from an integrated perspective with the goal being to develop a generic yet 
integrated Real Options-based decision-making framework that could be applied to 
address risks associated with DoD software acquisitions. 

D. HYPOTHESIS 

The traditional Real Options methodology, when enhanced and properly 
formulated around a proposed or existing software-investment, provides a framework for 
guiding software acquisition decision-making by highlighting the strategic importance of 
managerial flexibility in balancing the satisfaction of a customer’s requirements within 
the realms of the associated cost and schedule constraints. 

E. RESEARCH METHODOLOGY 

In an assessment of previous work conducted, it was discovered that while the 
Real Options approach had been largely studied independently from both technical and 
managerial perspectives, insufficient work had been done from a full spectrum “point of 
view” particularly, in the software engineering domain, i.e. the application of Real 
Options to both technical and managerial risks simultaneously in the context of a capital 
intensive software investment effort. Consequently, the approach proposed in this 
research explicitly identifies both the technical and managerial challenges facing software 
acquisition efforts and the formulation of a framework to address both concerns 
simultaneously rather than independently. The underlying premise for employing an 
integrated approach towards managing both technical concerns and managerial concerns 
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was based on the belief that both concerns should not be treated independently due to 
high correlation between technical and managerial issues. The following shows the 
research method undertaken: 

1. Tactics for Producing the Proposed New Method: This study, 
determines and establishes compliance with Real Options pre-conditions 1 
through 3 and explicitly infers that management has both the flexibility to 
develop Real Options to make mid course corrections and the wisdom to 
execute the Real Options when it becomes optimal to do so during the 
course of a software acquisition effort, thereby meeting the requirements 
established in pre-condition 4 & 5. 

The approach involved the development of a systematic methodology of 
uncertainty identification and risk quantification using the volatility of the 
returns on the software investment as a proxy to measure the risk facing a 
given software-related capital investment effort. 

The crucial issue of measuring and estimating the risks posed by 
uncertainty (pre-condition 3) to the software acquisition is addressed by 
employing a volatility estimation/refinement technique, based on 
Dempster-Shafer Theory (DST) or (Evidence Theory) which relies on 
“beliefs” and “plausibility” probability assignments. 

2. Methods to Substantiate New Method: To substantiate the results, a 
Real Options framework is developed explicitly showcasing the 
refinement of volatility estimation through the application of Dempster- 
Shafer Theory on Evidence. We attempted to validate the proposed 
approach using the software component (Euture Combat Systems 
Network) of the troubled multi-billion dollar U.S. Army Euture Combat 
Systems acquisition program as an example. Where feasible variables 
were derived and/or computed and in situations where data was 
unattainable, reasonable estimates along with the appropriate justification 
were utilized. 
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F. 


SIGNIFICANCE AND POTENTIAL IMPACT 


Software is about change. “Design for change”, “Invest with change in mind” are 
thus promoted as a value-maximizing strategy provided one could anticipate changes [8]. 
In most cases a software product is outdated even before it is delivered to the customer. 
More often than not, this is mostly due to changing customer requirements in response to 
changing business conditions. While arguments might be made by proponents and 
opponents that Moore’s Law^ holds or does not hold in this situation, the reality is 
change is inevitable in order to stay competitive in today’s environment. Therefore 
software change or evolution is inevitable. However, the time and mode of change itself 
is what is uncertain as it might be too early to predict, hence the need for a flexible risk- 
driven approach towards software-related capital investments both before and during the 
investment process. 

This phenomenon of continuous change has created the dilemma of huge cost and 
schedule overruns. In a yearly assessment and summary of major weapons acquisition 
programs published by the Government Accountability Office (GAO) it was reported that 
many, if not most, acquisition programs are experiencing cost overruns and schedule 
delays in their software development segments. For example in fiscal year 2006, the U.S. 
DoD spent as much as 30% ($12 billion) of its estimated budget of $40 billion for 
research, development, testing and evaluation on reworking software [7], a significant 
percentage when compared to the private sector software development. GAO also pointed 
out that, in the past five years, although “DoD has doubled its planned investments in 
new weapons systems from $700 billion to $1.4 trillion, this huge increase has not been 
accompanied by more stability, better outcomes or more buying power for the acquisition 
dollar” [7]. In an environment where program success has meant paying for massive cost- 
escalation, slipping plans years beyond their planned dates, making major cuts in the 
software functionality, the approach proposed in this study addresses these issues by 
taking a proactive approach to risk management by planning for an paying for risk up 
front. This is not to say that risk management strategies are not being adopted today, but 
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the failure of management to take a proactive approach towards risk management by 
employing what are believed to be “tactical” strategies, which unfortunately have not 
been proven to be successful 

G. SUMMARY OF MAJOR CONTRIBUTIONS 

This study contributes to the body of knowledge in the areas of software 
acquisition risk management by emphasizing the value of employing a proactive risk 
management approach. While there are several acquisition or decision-making 
frameworks currently utilized for decision-making in DoD software-related capital 
investments, at the time of this study we were not able to identify any situation where the 
Real Options approach had been explicitly applied to a single software acquisition 
program to guide the investment decision-making process; rather, we were able to 
identify instances where it had been proposed to employ Real Options within the context 
of IT portfolio management to manage a suite of IT investments. 

Specifically this study emphasizes the value of employing an agile approach in 
software-related capital investments/acquisitions by exploiting and borrowing on the 
concepts of the value of flexibility as currently adopted in Agile Development approaches 
to software development and incorporating this flexibility at the software acquisition 
level. We also emphasize the value of information realized by exposing the unknowns 
(uncertainties) much earlier in the software acquisition process and taking advantage of 
the embedded flexibility of adapt to the uncertainties. 

In this study we explicitly propose the use of Dempster-Shafer Theory on 
Evidence as a volatility estimation refinements mechanism, since volatility is a key input 
parameter needed for Real Options analysis. While volatility is just one of the parameters 
needed for Real Options analysis, it is the most difficult of all the parameters to estimate. 
We attempt to overcome the complexity of volatility estimation by proposing the use of 
Dempster-Shafer Theory on Evidence, a technique first proposed for application in the 
domain of sensor fusion. It is a mathematical theory of evidence, based on belief 

2 Moore’s Law describes the driving force of technological and social change by positing that 
advances in technology increases exponentially, by doubling approximately every two years. 
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functions and plausible reasoning, which is used to combine separate pieces of 
information (evidence) to calculate the probability of an event. We posit that it could be 
used to address both aleotoric and epistemic uncertainties inherent in software-related 
capital investments by ‘fusing” and reducing uncertainties to the maximum extent 
possible as they become revealed thereby facilitating a more accurate estimate of the 
risks propagated by uncertainty and allowing us to develop the appropriate option in 
response based on a more accurate volatility measure. 

Lastly, this strategic program management approach provides a means of 
overcoming the limitations associated with the spiral development process currently 
utilized in the Evolutionary Acquisition (EA) approach adopted in the DoD 5000 series 
acquisition directives by providing the much needed upfront risk management planning at 
the strategic level to complement the risk management approaches employed in the spiral 
development process. 

H. LIMITATIONS 

The Real Options approach requires us to compute two different values with some 
degree of confidence. These values are the cost of the software acquisition effort and the 
probability that circumstances would develop such that we would like to exercise the 
option. However due to the lack of detailed data on the Euture Combat Systems program 
at the level of granularity which we would have desired, we made several assumptions 
and provided justification as applicable. To determine the probability estimate we utilized 
modeling techniques along the lines of the three key assumptions made below and refined 
our estimate using Dempster-Shafer Theory on Evidence. This study is the first attempt at 
investigating the feasibility of utilizing Dempster-Shafer Theory on Evidence within the 
context of a software-related capital investment for volatility estimation, and while we 
were able to demonstrate its use, we did make some assumptions in the computation of 
our belief functions due to the lack of detailed data at our desired level of granularity in 
the estimates provided by the Cost Analysis Improvement Group and the Institute for 
Defense Analysis. 
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While we suceessfully applied the Dempster-Shafer Theory within the eonstraints 
of our assumption in our study, it is not without its limitations and eritics. Crities of the 
Dempster-Shafer Theory notably Judea Pearl, have argued that 

it is misleading to interpret belief funetions as representing either 
"probabilities of an event," or "the eonfidenee one has in the probabilities 
assigned to various outeomes," or "degrees of belief (or eonfidenee, or 
trust) in a proposition," or "degree of ignoranee in a situation” [23]. 

To overeome this eritieism we represent belief funetions as probabilities that a 
given proposition is provable from a set of other propositions [23]. 

1. ASSUMPTIONS 

As mentioned in the limitations seetion of this study, detailed data was not 
available at the desired level of granularity. While we were able to obtain data for the 
overall Future Combat Systems program, whieh ineluded eombined eosts of both 
hardware and software eomponents, we were not able to isolate software related eosts, 
i.e. eosts of the Future Combat systems Network (FCS-N) (software eomponent of the 
FCS program), from the overall cost data. Consequently, for the purposes of establishing 
a financial model of the Future Combat Systems program, we made three key 
assumptions. 

1. For cost purposes, it was assumed that the cost of the Future Combat 
Systems Network is equal to the costs of the overall Future Combat 
Systems program. What is accomplished by making this assumption is 
rather than picking an arbitrary value as the cost of the Future Combat 
Systems Network, we treat the overall Future Combat Systems program as 
consisting of software alone. 

2. We assume the estimated future benefits (Asset Value) of the Future 
Combat Systems Network, while unknown, translated into a positive value 
under traditional Net Present Value techniques (i.e. benefits outweigh 
costs). 
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3. We assume the independent assessments provided by the Cost Analysis 
Improvement Group and the Institute of Defense Analysis ineludes belief 
assignments based on masses of evidenee during the application of 
Dempster-Shafer Theory. That is, the executive or decision maker is 
provided not only with a raw set of the risk factors of requirements creep, 
integration risk, performance risk, but with additional measures: belief in 
the estimation of the risk factors and certainty of the estimation during the 
decision making process. 

J. DISSERTATION ORGANIZATION 

This dissertation is organized around seven chapters, containing results from our 
Risk Simulator provided by Real Options Valuation Inc., and a list of references and 
bibliography. 

Chapter II discusses the relevant background material and highlight weaknesses 
and gaps in the current state of knowledge by discussing key areas of concern and 
limitations of current investment valuation models. We conclude this chapter by 
providing the reader with some insights into real options theory. 

Chapter III presents detailed insights in the area of uncertainty, a key component 
necessitating the need for the real options approach proposed. Specifically we categorize 
the uncertainties facing program mangers into technical and managerial uncertainties and 
further expand on the details associated with each category of uncertainty. 

Chapter IV addresses the issue of estimating the volatility of the returns of a 
software investment based on the prevalent risk factors introduced by uncertainty. We 
also depict how the proposed Dempster-Shafer Theory based on the Dempster’s rule of 
combination is used to combine evidence from different sources. 

Chapter V culminates in the formulation of the proposed Real Options framework 
that could be applied to software-related capital investments. 
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Chapter VI validates the proposed framework by exploring the eoneeptual 
applieation of our proposed framework to the U.S. Army’s Future Combat System 
Network (software eomponent) aequisition program. 

Chapter VII presents a summary of our findings, eontributions and 
reeommendations for future work. 
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II. ASSESSMENT OF PREVIOUS WORK 


The proper measurement of the success of software investments requires 
more than compliance with models of technical perfection. Technical 
excellence of programming code is highly desirable but clearly 
insufficient. Nowadays the most profitable and popular software is 
notorious for its bugs and glitches. Economic utility and independent 
measures of customer satisfaction must be the ultimate arbiters of all 
judgment about the utility of the software. 


(Strassmann, 1997 p.l47) 

A. SOFTWARE INVESTMENTS BACKGROUND 

The software investment process is a complex process plagued by a myriad of 
activities which could be broadly categorized into technical and managerial issues, both 
of which are inextricably linked. Understanding the risks associated with software 
investment activities is therefore an important consideration for a project manager and 
must be captured in an investment strategy designed to accomplish the goals of the 
investment effort and ultimately executed in an acquisition plan. Therefore given its 
complexity, a software project manager must be well equipped and postured to deal with 
a large synergy involving players ranging from the project manager him/her self to the 
end users [11]. This is accomplished through the development and use of a systematic 
decision-making approach that would bridge the gaps between technical and managerial 
activities in the form of a framework which would enhance decision-making associated 
with choosing the appropriate courses of action in a complex, uncertain, or conflict- 
ridden situation. 

Technical activities may range from requirements specification to implementation 
while managerial activities may range from planning to risk management with the early 
and unambiguous definition of software requirements being key to the investment 
process. Regardless of the activity, both of these categories of activities presents 
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challenges in the form of uncertainties necessitating the need for a methodology to 
identify uncertainty, estimate and ultimately manage risk to further guide the investment 
proeess. 

From a teehnieal perspeetive, the software arehiteeture is viewed as the earliest 
design artifaet, whieh realizes the requirements of the software system [24] with the 
main focus of the software architecture being to provide a “high level” abstract design 
plan with just enough detail to manage eomplexity, to provide the overarching framework 
and guidanee for detailed design. Thus software arehiteeture ean be viewed as: 

the manifestation of the earliest design decisions, which comprise the 
arehiteetural structure (i.e., components and interfaces), the arehitectural 
topology (i.e., the arehiteetural style), the arehiteetural infrastructure (e.g., 
the middleware), the relationship among them, and their relationship to the 
other software artifacts (e.g., low-level design, testing etc.) [24] 

However, in virtually all software engineering activities, the software architecture 
is usually driven by eustomer requirements, with several iterations oecurring between 
requirements and arehitectural design until a preliminary design is finalized, from which 
a baseline for detailed design ean be established. Thus we ean reasonable claim that 
customer requirements, as identified by an operational need reflect the earliest 
manifestation of the software investment effort which precedes the software arehiteeture. 

B. INVESTMENT STRATEGIES 

Traditionally speaking there are two main approaehes whieh guide a software 

investment strategy; eustom development of the software and acquisitions based on the 

use of COTS products, otherwise known as the “build vs. buy” phenomenon. Eaeh of 

these present their pros and eons and need to be carefully evaluated before ehoosing one 

approach over the other. This is highlighted in Figure 3 below in whieh differences are 

depicted between both approaches in the form of “time to realization of benefits” [20]. 

However it must be noted that the study conducted in [20] lacked a elear delineation 

between software and the assoeiated infrastructure (hardware etc) within the investment 

framework. Speeifically the research in [20] highlights that in the COTS aequisition 

approach, the organization has the option of spending an amount of money (K) to aequire 
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the IT asset. At any point of time (t) during an interval T, K is known with certainty; 
however, future changes in K are uncertain. After the asset is acquired, the organization 
starts receiving a set of cash flows (C) representing the differential benefits derived from 
acquiring the IT asset. 


ITAcquisition Project 
Wait 

0 


Invest 

K 


1 


Receive C 






IT Development Project 


Wait 


0 


Start Investing 
_ 

t 


Invest (l£Tni) 


Receive 

V 



Figure 3. Two Types of IT Investment Projects (From: [20]). 

Given that both the cost and the benefits are uncertain, it might be better to wait 
before making the investment. Furthermore, if the cost of a particular IT asset decays 
over time, there is an additional incentive for waiting before acquiring the asset. 
However, benefits also decrease with time because waiting will reduce the length of 
period in which the organization will be able to receive the benefits associated with the 
investment. Therefore, both elements have to be taken into consideration for making an 
optimal decision [20]. 

In the development approach, the asset is not acquired instantaneously; rather, it is 
the result of a development project having an uncertain duration (r) in which the firm 
keeps investing at a rate that is less than or equal to a maximum investment rate (Im). 
Only until the project is completed and the remaining cost (K) is zero, the firm receives 
the underlying asset (V) [20]. 
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While the software investment approaeh is somewhat analogous to the IT 
investment approaeh depicted above, this paradigm however, is not completely 
representative of all software engineering investment efforts, as software development 
usually takes a considerable amount of time, and the benefits might not be obtained until 
the project is completely finished. It is however possible to start enjoying benefits of the 
software investment, albeit partially when modern software processes such as 
incremental development which support custom software development or COTS 
development within an evolutionary context are utilized. 

C. WEAKNESSES AND GAPS IN STATE OF THE KNOWLEDGE 

In general one of the key challenges facing software and its acquisition are 
requirements issues. This dilemma could include scenarios of inadequate or insufficient 
thought or effort going into the upfront investment decision to outright ignoring the 
future evolution of the software investment effort under consideration. The theoretical 
foundations of strategic planning offer a focus on “long-term” thinking, and when we 
think “long-term” in software engineering, evolution readily comes to mind. An 
evolutionary frame of mind should therefore form the underlying premise under which 
all software-related capital investment are made with emphasis being on the keyword 
“capital”, where future operational requirements would serve as the driver of evolution. 

While options theory has been largely studied and applied to decision-making in 
the area of Information Technology (IT) portfolio management and to the various 
software artifacts, e.g. software architecture [24], little work if any has been done to study 
its application to all the decision-making activities (both technical and managerial) within 
a single framework to manage a single software investment as opposed to a portfolio of 
software investments. A holistic approach towards managing technical and the associated 
managerial issues is therefore needed to keep up with the advances in the technical 
aspects of software engineering and to also better manage risk. This dilemma is 
highlighted in several reports published by the U.S. Government Accounting Office 
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(GAO), and in a white paper on effeetive technology investment strategies [13], the 
author calls for the need for a more rigorous scientific process for developing a 
company's technology strategy and technology investment, in lieu of the trial-and-error 
process. 

Thus given these gaps in the current state of the knowledge, software executives 
and program managers need better decision making tools and methodologies to guide 
their software investment decision-making activities. To accomplish this, the DoD 
proposed the Evolutionary Approach to software acquisition which we discuss below 
along with the technical and managerial perspectives. 

1. Evolutionary Perspective 

In May 2003, the U.S. Department of Defense (DoD) promulgated revised 5000 
series acquisition directives and instructions that mandated an evolutionary approach 
known as Evolutionary Acquisitions (EA) (Eigure 4) which relied on the spiral 
development process. 
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Figure 4. Evolutionary Acquisition and Spiral Development (From: [12]). 


The strategy employed within the EA construct was to deliver capabilities in 
incremental fashion recognizing the up-front need for future capability improvements 
closely integrate managerial and technical decision-making. However, the overarching 
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limitation of this approach could be narrowed down to the ehosen development proeess - 
the spiral development process. The spiral development proeess assumes the end-state 
requirements are known at the ineeption of the development proeess [12], In a study 
eondueted by RAND, it was diseovered that EA programs required eonsiderable 
additional up-front management planning and engineering workload and the budget 
sourees to support them [12], The spiral development proeess is a risk-driven 
development approaeh whieh eonsists of four main phases namely: determining 
objeetives/alternatives, risk analysis, development and planning. The phases are 
iteratively followed one after the other building progressively on the first iteration until a 
eomplete software produet is built. Of the four phases, the risk analysis phase is the most 
important beeause the projeefs sueeess is highly dependent on the ability to identify and 
resolve risk. Risks are eontinuously diseovered and high-priority risks drive the 
development proeess. However, besides the obvious limitation of the spiral approaeh 
being a somewhat costly approach, we believe that risk management should be a factor 
that is addressed much earlier in the software engineering proeess - at the aequisition 
level, during the investment deeision making activities prior to the eommitment to 
aequire and or develop a software system. The appropriate risk mitigation/reduction 
strategies should be employed mueh earlier in the software investment/aequisition 
proeess. 

Furthermore the RAND study [12] also determined that EA programs using the 
spiral development proeess needed to focus on capability mission objectives rather than 
traditional teehnieal requirements and in the analysis of five ease studies all of the case 
studies pointed to the eonelusion that the eapabilities and requirements definition and 
management proeesses eontinued to remain as major ehallenges in all EA programs. 

2. Technical Perspective 

Relative to other engineering fields, software engineering is still very mueh in its 
infaney. Computer seienee, management seienee and some other fields have greatly 
infiueneed the theoretieal foundations for software engineering, however the 
standardization and the lack of a clean theory to make software design deeisions 
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continues to compound the problem of software investments. While the guiding 
principles of software design which center on abstraction, information hiding, 
localization, modularity, uniformity, conformability, completeness serve to ensure that a 
useful software product is developed, by establishing a foundation for which the overall 
goals of correctness, flexibility, robustness, efficiency and reusability in software design 
are achieved, the challenge remains, one of correlation of these idiosyncratic principles 
from the perspective of options theory [25]. The work of Kevin Sullivan [25], also 
proposes an options-based interpretation in software design activities by connecting 
options theory to software design in two steps: 1) viewing software design as a value 
creating opportunity to make irreversible capital investments in software assets of 
uncertain value with the architecture, documents, program generators, and information 
hiding interfaces being examples of assets and 2) rationalizing decisions about whether 
and when to make such investments by appealing to options theory and viewing 
investment opportunities as call options [25]. The uncertainty in the value of investing in 
a software asset makes this problem a prime candidate for the application of a decision 
making theory such as options theory to help guide the overall investment process and 
consequently software program managers are faced with the problem of overcoming 
some rigid dictums in software engineering since there are no reliable proxies to guide 
optimal investment strategies. 

Thus software artifacts are and should be treated as software assets due to the 
particular value they add to the overall investment process. Assets can be software 
components, objects, software requirement analysis and design models, domain 
architecture, database schema, code documentation, manuals, standards, test scenarios, 
and plans and more often than not were either created by prefabrication i.e. the assets 
were developed with posteriori reuse thought process (designed for reuse). An effective 
investment strategy should not only address the possible uncertain future evolution of the 
software but also factor in an effective reuse strategy that could possible support the 
future evolution of the software. This of course introduces the problem of architectural 
stability and the ability of the architecture to accommodate the changes, a problem which 
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Bahsoon [24] attempted to address in his study through the proposal of an “Arch 
Options” model. 

Evolution of software systems and the associated ability of the architecture to 
support evolution was the key theme of [24] in which a “Arch Options” model was 
proposed to address architectural stability in the face of changing requirements in an 
evolutionary context. The “Arch Option” approach provided guidelines on eliciting the 
likely changes in requirements and relating architectural decisions to value. 

The “Arch Option” framework also accounted for the economic ramifications of 
the change on the structural (e.g., maintainability) and behavioral (e.g., throughput) 
qualities of an architecture and on relevant business goals (e.g., new market products). In 
[24], Bahsoon acknowledges that the valuation of the architectural potential to the change 
is a multi-perspective problem and attempts to tackle the problem by proposing a 
valuation “point of view” framework for quantifying the options from different 
perspectives. However their work stops short of identifying ways to manage the valuation 
under this framework, such as identifying the dimensions, which are critical for 
understanding architectural stability, prioritizing and weighting the valuation of these 
dimensions, managing conflicts, and reconciling the options results. This is necessary to 
provide a sound comprehensive valuation, which takes into account the various valuation 
points of views. Consequently our proposed approach would embrace a “holistic” 
approach towards uncertainty identification that would be utilized much earlier on in the 
software investment process to manage and reduce uncertainties and we then employ 
Dempster-Shafer Theory of Evidence based on subjective probabilities as a mechanism 
to further refine our risk estimates associated with the uncertainties. (See Chapter 4 for a 
detail discussion of the Dempster-Shafer Theory of Evidence.) 

3. Managerial Perspective 

Economic realities demand that organizations leverage their existing and future 
software investments to create a competitive advantage, be it in an operational capacity or 
support capacity to the war-fighter. More often than not, this involves the delicate art of 
balancing operational needs which might be compounded with uncertainty with costs and 
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schedule limitations. To aceomplish this feat, deeision making tools are needed and are 
either ereated on an ad-hoe basis or well established deeision-making aids utilized. We 
examine two of the more popular teehniques below 

a. Decision Tree Analysis 

Decision tree’s are popular decision support tool used as a visual and 
analytical means for caleulating conditional probabilities to help identify the strategy 
most likely to produce the optimal solution goal. 



Figure 5. Sample Decision Tree (From: [19]). 

It is a ehronologieal representation of the deeision proeess [16] and 
utilizes a network of nodes. It provides a prescriptive approach to decision-making by 
allowing the deeision maker to seleet exaetly one alternative from a set of possible 
deeision alternatives in situations of uneertainty regarding the future. Nodes indieate 
deeision points, ehanee events, or branch terminals with the root node representing the 
first set of decision alternatives, q represents the relative outcome probability, or 
uncertainty, associated with eaeh ehanee event. 

b. Utility Theory 

Utility theory is an attempt to infer subjeetive value, or utility, from 
ehoiees and it ean be used in both deeision making under risk (where the probabilities are 
explieitly given) and in deeision making under uneertainty (where the probabilities are 
not explicitly given) [22]. With its philosophy based on the doctrine of utilitianism, its 
underlying assumption is that the decision maker always chooses the alternative for 
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which the expected value of the utility (EXPECTED utility) is maximum [18]. In other 
words, expected utility eould more preeisely be ealled "probability-weighted utility 
theory”. It involves the construction of a utility function based on assignment rules in 
whieh a utility is assigned to eaeh of the possible (and mutually exelusive) consequences 
of every alternative, depending on the individual preferenees of the decision maker. 
While there are several methods for constructing utility functions, the best-known method 
is based on indifferenee judgments of the decision maker about specially constructed 
alternatives. Objeetive utility is the dominating approach in risk analysis, and the 
common way to measure risk is to multiply "the probability of a risk with its severity, to 
eall that the expeetation value, and to use this expeetation value to compare risks” [21]. 

It is essential to observe that Decision-tree analysis (DTA) moves the 
traditional Net Present Value analysis which we would be discussing in the next section, 
one step forward by allowing the possibility of alternative states of nature. Eurthermore, 
expected utility maximization is only meaningful in comparisons between options in one 
and the same deeision, with some of the elearest violations of this basie requirement 
found in risk analysis where expected utility ealculations are often used for comparisons 
between risk factors that are not options in one and the same deeision (deeision theory). 

D. INVESTMENT VALUATION METHODS 

Estimating the value of a software investment effort is a partieularly challenging 
task beeause there are many factors that affect the payoffs and eosts of the investment 
effort. While there are several teehniques for valuation, valuation approaehes could be 
categorized into three mainstream approaehes namely, the market approaeh, ineome 
approach and the eost approaeh [14]. The market approaeh looks at eomparable assets in 
the market place and their corresponding prices and assumes that market forees will tend 
to move the market priee to an equilibrium level, the income approach looks at future 
potential profit of the asset and attempts to quantify, forecast and discount these net free 
cash flows to a present value, and the cost approach which examines the costs that would 
be incurred if the asset under consideration were to replaced or ereated from seratch [14]. 

In the field of finance, valuation is the process of estimating the market value of a 
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financial asset or liability with the Capital Asset Prieing Model (CAPM) being the most 
popular teehnique. 

From the perspeetive of a software investment, valuation can be defined as the 
proeess of estimating the present and future market value of software investment (asset). 
However all these valuation approaches share one thing in eommon; they rely on the 
Diseounted Cash Flow (DCF) teehnique whieh eontinues to be the most popular 
investment valuation techniques. We further expand on this teehnique in the next seetion. 

1. Discounted Cash Flow (DCF) Model 

The DCF method is an approaeh used to valuation, whereby projeeted future cash 
flows are “diseounted” at an interest rate or rate of return that refleets the perceived 
riskiness of the eash flows. The DCF model has two eomplementary measures; Internal 
Rate of Return (IRR) and Net Present Value (NPV), with the Net Present Value being the 
single most widely tool used for large investments made by eorporations. 

Net Present Value (NPV) measures the exeess or shortfall of eash flows, in 
present value (PV) terms, basically a cost benefit analysis methodology and is simply 
eomputed by subtracting costs from benefits, where benefits equal the sum of the present 
value of future cash flows after taxes, diseounted at some market risk-adjusted costs of 
capital and costs equal the present value of investment eosts diseounted at the risk free 
rate or reinvestment rate [14]. It is an indieator of the value an investment adds to a firm. 
Deeisions making under this concept are based on the sign of the eash flows. In financial 
theory, if there is a choice between two mutually exclusive alternatives, the one yielding 
the higher NPV should be selected [17]. The general formula for eomputing NPV is 
given as follows: 



-Co 


where 

t - the time of the eash flow 
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N - the total time of the project 

r - the discount rate (the rate of return that could be earned on an investment in the 
financial markets) 

Ct - the net cash flow (the amount of cash) at time t (for educational purposes, Co is 
commonly placed to role as the initial investment). 

Table 2 below showcases the decision making criterion associated with NPV. 


If... 

It means... Then... 

NPV>0 

The investment would add value to the 
firm 

The project may be accepted 

NPV<0 

The investment would subtraet value 
from the firm 

The project should be rejected 

NPV = 0 

The investment would neither gain nor 
lose value for the firm and does not 
mean that the investment is expeeted 
to break even. 

We should be indifferent in the decision 
whether to accept or reject the project. This 
project adds no monetary value. Decision 
should be based on other criteria, e.g. 
strategic positioning or other factors not 
explicitly included in the calculation. 


Table 2. Net Present Value Decision Making Criteria (From: [17]) 


Despite the wide use of the NPV technique, it has one flaw - its inability to 
explicitly account for managerial flexibility. Thus we explore the RO approach, which 
takes into consideration a key concept: present day management flexibility to make 
strategic decisions that have a long-term strategic impact, a concept that is noticeably 
absent in other valuation methods such as Discounted Cash Flow (DCF) and Net Present 
Value (NPV) [26]. 

E. REAL OPTIONS METHODOLOGY 

The Real Options approach offers a means of capturing the flexibility of 
management to address uncertainties as they are revealed. With its history and theory 
embedded in financial theories, the key valuation concept of (financial) options theory is 
that an option can be priced based on the construction of a portfolio of a specific number 
of shares of an underlying asset, and that one can borrow against the shares at a riskless 
rate to replicate the return of the option in a risk-neutral world [19]. An option gives its 
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holder the right, without the obligation, to aequire or dispose of a risky asset at a set 
strike price within a speeified time period [27]. If the market eonditions are favorable 
before the option expires, the holder exereises this right, thus making a profit. Otherwise, 
the holder lets the option expire. This asymmetric nature of options gives them real 
economic value [27]. 

When extended to real assets, a Real Option could be defined as a systematic 
approach and integrated solution using financial theory, economic analysis, management 
science, decision sciences, and econometric modeling to valuing real physical assets, as 
opposed to financial assets, in a dynamic and uncertain business environment where 
business decisions are flexible in the context of strategic capital investment decision¬ 
making, valuing investment opportunities and project capital expenditures [26]. In 
essence a real option is a flexible arrangement that acknowledges the ability of 
management to make decisions to counter against some unforeseen situations, in this case 
being uncertainties in software engineering. However creating the options, acquiring the 
options and managing the options over time and to realize the full potential value are 
some of the challenges that need to be addressed. 

The Real Options approach is able to overcome the limitations of traditional 
valuation techniques by utilizing the best features of traditional approaches and extending 
their capabilities under the auspices of managerial flexibility. In a real options view, 
uncertainty is the randomness of outcomes from a software investment decision. Real 
options are implicit or explicit capabilities created for real assets [28] that provide the 
software manager with time-deferred and flexible choices (options) regarding future risks 
or changes of the software and could explicitly address the issue of software investment 
choices for future capabilities. It assumes that managers develop a level of foresight 
sufficient to invest in resources/techniques and processes with ‘options’ characteristics 
that provide implicit or explicit claims on future opportunities and generate flexibilities 
for future investments or changes [28]. Through these capabilities, the software manager 
may choose to adjust, reduce, increase, or abandon the investment in the future, thereby 
stabilizing returns from these assets. In other words, it analyzes how software managers 
can lay claim to future rent-generating capabilities through investment in options. The 
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real options approach helps to structure the project as a sequence of managerial decisions 
over time, clarifies the role of uncertainty in project evaluation and allows us to apply 
models that have been developed for valuing stock options to project investments [20] 
(Bodie and Merton 1999), and our study seeks to develop a RO framework to explicitly 
estimate risk using Dempster-Shafer Theory of Evidence and Dempster’s rule of 
combination to guide the decision making process. Using the options logic necessarily 
entails a rigorous analysis of the software investment, their uncertainties, and costs of 
creating the options, all of which contribute towards a greater understanding of the 
strategic role for the software. We shall begin the formulation of our framework by 
visiting and addressing the issue of uncertainty in the next chapter. 
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III. ADDRESSING UNCERTAINTY 


As we know, there are known known’s; there are things we know we 
know. We also know there are known unknowns; that is to say we know 
there are some things we do not know. But there are also unknown 
unknowns—the ones we don't know we don't know. 

(Donald H. Rumsfield, Seeretary of Defense, 2002) 

A. INTRODUCTION 

Uncertainties permeate virtually every phase of the software investment process 
from procurement decision-making, requirements specification, software development 
and implementation, to the eventual evolution of the software. This sentiment is best 
echoed by Ziv’s uncertainty principle of software engineering in which he posits that 
uncertainty is inherent and inevitable in software development processes and products. 
These uncertainties usually present themselves in various forms ranging from 
changing/incomplete requirements, insufficient knowledge of the problem domain to 
decisions related to the future growth or evolution of the software. As identified earlier 
on in Chapter I, the presence of uncertainty is one of the pre-conditions necessary for the 
application of Real Options (RO). In this chapter, we attempt to capture the uncertainties 
that appear in the software investment process and how they impact decisions associated 
with software investments. 

First, it is important to understand what the term “uncertainty” means. The word 
‘uncertainty’ as derived from Webster’s dictionary defines uncertainty as a situation 
where; 

The current knowledge available may range from falling short of certainty 
to an almost complete lack of conviction or knowledge especially about 
the outcome or result. 

When translated literally, the key phrase “may range from falling short of 
certainty to an almost...” hints at some element of probability and insinuates the presence 
of some degree of risk, which is either introduced or mitigated within the context of 


29 



uncertainty. Software engineering, regardless of process, programming language, or 
domain, involves signifieant deeision making on the part of the software manager due to 
the myriad of activities involved in the software engineering proeess and the uncertainties 
that surround these activities. The driving foree of the uncertainty of future software 
evolution to meet either future organizational needs or operational challenges also 
increases the degree of uncertainty associated with the software investment deeision 
making process. As a result, uncertainty introduces and drives risk. We must however 
emphasize that uncertainty should not be confused with risk as there is an important 
distinction between the two. Risk is something one bears and is the outcome of 
uncertainty, as uncertainty is either resolved through the passage of aetion or left 
unattended due to inaction [14]. The risks associated with the acquisition of the software 
need to be identified and analyzed very early on in the decision-making process, and an 
approach to mitigate the high-priority risks must be ineorporated into a software 
aequisition plan. Therefore, the responsibility lies with the software manager to identify, 
manage and eliminate the sources of uncertainty by developing a risk management plan 
so as to identify risks at the earliest possible time, adjust the acquisition strategy to 
manage high-priority risks, and implement a risk management proeess to manage risks 
throughout the acquisition life cycle [40]. 

B. CATEGORIZATION OF UNCERTAINTIES 

In addressing the issue of uncertainty, we focus on uncertainty from three points 
of view: the uncertainties associated with the software investment decision-making 
process, uncertainties with the software engineering process and uneertainties associated 
with the software product itself 



Figure 6. Taxonomy of Uncertainty (From: [29]). Classification of 

uncertainty into epistemic (reducible) and aleatoric (irreducible). 
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According to the scientific body of knowledge on uneertainty in aeronauties and 
astronauties, uneertainties exhibit themselves in either of two forms as depieted in Figure 
6 on the previous page. Epistemie uneertainties are eonsidered to be reducible 
uneertainties while aleatorie uneertainties are eonsidered to be irredueible uneertainties. 
While epistemie uneertainty deals with our laek of knowledge, laek of information and 
our own and others’ subjeetivity coneeming an issue, aleatorie uneertainties, on the other 
hand, deals with the randomness (or predietability) of an event due to variability of input 
or model parameters when the eharaeterization of the variability is available [29]. In other 
words, an aleatorie uneertainty is an inherent variation associated with the physieal 
system or the environment. Both epistemie and aleotoric uncertainties are interwoven and 
form the general framework which uncertainties fall into and also form the framework 
from whieh we would be addressing uneertainty in this study. 

C. INVESTMENT DECISION MAKING UNCERTAINTIES 

Several faetors drive uneertainties in the software investment deeision making 
proeess. To determine these uneertainties, we first identify the pertinent activities leading 
to and associated with a software investment effort and establish an overarching 
framework. The three major aetivities as identified in Figure 7 below are the software 



Figure 7. Software Investment Process. 
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investment decision-making activity during which a decision is made to procure a 
software system, the associated development activities of the software and 
implementation related activities. 

The overwhelming uncertainty surrounding the software investment decision¬ 
making activity is determining what the scope of the acquisition effort would be, the 
costs and determining the most appropriate acquisition strategy. Possible acquisition 
strategies range from purchasing the desired capability outright from commercial vendors 
and then customizing it to suit the customer’s needs, building a custom solution from 
scratch, or employing a “hybrid” approach that includes a combination of both custom 
development and purchasing of the components from commercial vendors. These are 
examples of Real Options strategic paths, that is, options to undertake different courses of 
action (analysis of alternatives). Under normal conditions, the investment decision¬ 
making activity phase is initiated upon the conclusion of an “operational needs” 
assessment to justify the needs for investing in the capability by the using organization 
(Army, Navy, Air Force, and so forth). 



Figure 8. Key Software Investment Decision Making Activities. Investment 

decision making activities include choosing between a “Build”, “Buy” 
or “Hybrid” acquisition strategy each of which has uncertainties 
associated with them which could either be epistemic or aleotoric in 

nature. 
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In the case that the operational need is conceptual in nature, trade studies should 
be conducted to either prove or demonstrate the concept before an investment decision is 
made and before funds are committed to the effort. The typical “high level decisions” 
associated with the investment decision making activity are depicted in Figure 8 above. 

The investment decision making activity drives the development phase, which 
centers on the technical activities associated with developing the solution to meet the 
operational need once an investment decision is made and an acquisition strategy 
selected. This leads to the implementation activities phase whose end result centers on the 
delivery and fielding of the software as well as supporting the maintenance and possible 
evolution of the software. In this study, we limit our focus to studying the uncertainties 
associated only with the investment decision making process due to our belief that 
uncertainties associated with the investment decision-making activity drives the overall 
success of the acquisition program, hence the need to address the uncertainties early on in 
the investment process. We now proceed with developing an in-depth methodology for 
identifying and addressing uncertainty early on in the software investment process 
through an uncertainty elicitation phase. 

D. UNCERTAINTY ELICITATION 

Just as a formal requirements elicitation phase exists in traditional software 
engineering processes, we propose the introduction of a formal and distinct “uncertainty 
elicitation” phase as part of the software investment decision-making process. This phase 
would not include members of a typical requirements team, but would work in tandem 
with them, to play the “devils advocate” by identifying and documenting uncertainties as 
they are revealed in the forces driving the need for the software, the acquisition strategy 
and its ultimate implementation. Implementing an uncertainty elicitation phase would 
facilitate the identification of uncertainties very early in the acquisition process, and steps 
could be taken to either refine the requirements to address the uncertainties or strategic 
options identified to mitigate the risks posed by the uncertainties. In Figure 9, we expand 
on the “Software Investment Decision” component of Figure 7 and propose a 
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methodology that could be used to capture and address uncertainties at the inception of a 
software-related capital investment through the introduction of an explicit step to allow 
for uncertainty elicitation. 



B access re j^tartes 


Figure 9. Uncertainty Elicitation. The “usable requirements” are elicited at 

a very high level and could help guide the investment decision making 
or selection of an appropriate acquisition strategy. 

Once all these activities have been accomplished and a decision is made to 
proceed with the software investment effort, we now proceed to the development 
activities phase and attempt to develop an in-depth methodology for uncertainty 
identification in the remainder of this chapter. Development activities center on all the 
tasks and activities that are associated with developing the software product. Uncertainty 
appears in various phases during the development process and we attempt to identify and 
address them. 
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E. SOFTWARE ENGINEERING UNCERTAINTIES 


As software engineering uncertainties are present at all stages of the software life 
cycle, these uncertainties which are treated implicitly should be handled explicitly by 
developing a proactive strategy to either mitigate or hedge against the uncertainty. The 
challenge remains on how to identify and capture uncertainties explicitly, reason in the 
face of uncertainties (analysis, trade-offs) and ultimately manage the uncertainties. 
Accordingly, Lehman’s proposal of the basic uncertainty principle of computer 
application [41], which stipulates that the outcome in the real world of software system 
operation is inherently uncertain with the precise area of uncertainty not knowable, 
clearly highlights the inherent difficulties of addressing uncertainty. The extent to which 
a degree of satisfaction is reached that the operational system meets its intended 
requirements is ultimately determined by the software professionals’ judgment, action, 
and inaction. Thus, neither developers nor users can fully know system properties, 
therefore solving these problems in the face of these uncertainties to produce solutions 
that satisfy, in spite of not knowing what it is that they satisfy, is the heart and essence of 
software engineering [41]. 

These uncertainties have given rise to risk and decision analysis methodologies 
and paradigms, such as modeling and simulation to model the unknown and delaying 
certain decisions for as long as possible, that are targeted at reducing epistemic 
uncertainties by increasing one’s knowledge of the problem at hand. Since the epistemic 
uncertainties facilitate the introduction of aleatoric uncertainties in the end product, an 
approach is needed to vastly reduce epistemic uncertainties in order to reduce the 
corresponding aleatoric uncertainties which are propagated in the software engineering 
effort. 

In an attempt to provide clarity, we revisit the definition of software engineering, 
which we loosely define as the “the set of activities leading to the development of a 
software product.” We categorize these activities into two interwoven but distinct phases 
to further highlight and delineate the uncertainties which arise in each of the two phases: 
The first phase is the "program management” phase and the second phase is the 
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“technical” phase. The activities in the first phase are heavily slanted toward managerial 
activities which involve significant decision making on the part of the software program 
manager regarding the technical activities surrounding the software development effort. 
We highlight the development activities in Figure 10. 



Figure 10. Development Activities & Software Engineering Activities. 


F. MANAGERIAL PERSPECTIVE 

The managerial phase serves to provide program management guidance to ensure 
the software product is developed within cost and scheduled constraints. From a 
managerial perspective, the five major constraints of people, time, functionality, budget, 
and resources (excluding people) [39] form the basis of the uncertainties which plague 
the software program manager. These five constraints could be summed up into two 
major uncertainties, cost estimation and scheduling uncertainties (Figure 11). 
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Figure 11. Managerial Activities. 

The variables of eost, time, and funetionality (seope) are intrinsieally linked in the 
managerial activities centered around cost and schedule estimation. When time and cost 
are the essence of a project, then functionality becomes defined and is no longer a 
variable while on the other hand if functionality is defined, then cost and time are 
defined. Therefore if any one of these points becomes fixed, the other two points become 
controlled [42]. Maintaining a perfect balance between these three variables would lead 
to more accurate estimate and reduce the likelihood of cost overruns. 

1. Cost Estimation 

The accurate prediction of software development costs is a critical issue during 
the acquisition decision-making process. Cost estimation uncertainties arise due to the 
intricacies involved in determining the complexity, amount of work (size) and associated 
costs. Statistically speaking, escalations in the dominant cost of Tabor’ have led to 
spiraling project costs as project scope and schedule escalates resulting in outright 
cancellations and budget overruns. Productivity of the team can also be difficult to 
predict. For example, in previous research, the case of employee productivity has been 
likened to that of returns on financial stock because productivity may vary by orders of 
magnitude, a situation which is analogous to the theory of financial instruments, where 
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empirical evidence suggests that the root cause of uncertainty is variance in the return 
rates, not in the base stock values [34], 
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Figure 12. Cone of Uncertainty (From: [37]). The horizontal axis contains 
common project milestones such as Initial Concept, Approved Product 
Dellnition, Requirements Complete while the vertical axis contains the 
degree of error that has been found in estimates created hy skilled 
estimators at various points in the project. 

While there are several software cost estimation methods available such as 
algorithmic methods, top-down method, and bottom-up method, to name a few, no one 
method is necessarily better or worse than the other and more often than not the strengths 
and weaknesses of the estimation methodologies are often complimentary to each other. 

Therefore to mitigate these uncertainties, an accurate estimation of a variable 
phenomenon must include the variability in the phenomenon itself In other words, costs 
should be determined for uncertainties within an estimate, hence our proposed Real 
Options approach. The variability in these factors contributes to the variability of project 
estimates and as these sources of variability are further investigated and pinned down, the 
variability in the project diminishes, thereby diminishing the variability in the project 
estimates [37]. 
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This phenomenon otherwise known as the “Cone of Uneertainty” and is best 
illustrated as depleted in the Figure 12 above. Sinee uneertainty-redueing deeisions are 
usually not yet applied at the beginning of a projeet, the eone is always wider at the 
beginning of a projeet. The eone begins to narrow as uneertainty-redueing deeisions are 
implemented. The optimal desired seenario for the projeet manager is to make the eone as 
narrow as quiekly by applying uneertainty-redueing strategies whieh produeed the 
desired outeome. This would require a diseiplined and eomprehensive approaeh of 
assessing and the eost estimation uneertainties and determining the projeet risk faetors. In 
the ease that eertain eosts are unknown, a justifiable eost should be ineluded in the form 
of a eontingeney amount. 

2. Scheduling 

The seheduling aetivity is inextrieably tied to the projeet’s teehnieal baseline and 
is essential to developing a eost estimate for the overall effort. It usually starts with the 
development of a work breakdown strueture (WBS) whieh represents a hierarehieal set of 
independent tasks with the preeedenee relationship that exists among tasks defined. Input 
from the eost estimation phase is usually required in the WBS and the final produet is 
presented in the form of preeedenee networks and Gantt eharts in whieh the eritieal path 
is identified. 

Teehniques sueh as the PERT model (Program Evaluation and Review 
Teehnique) were developed to address uneertainty in the estimation of projeet 
parameters. However, the main problem with PERT is that it gives aeeurate results only 
if there is a single dominant path through a preeedenee network [38]. In order to deal 
with an uneertain environment it is neeessary to eonsider setting and employing a 
eertainty threshold before a projeet is started, and ensure that it is monitored throughout 
the exeeution lifetime of the software projeet and reseheduling is performed if the 
eertainty threshold is breaehed [35]. In situations where a single path is not dominant, 
PERT usually provides overly optimistie results henee the need for a Monte Carlo risk 
simulation approaeh, whieh would repeatedly sets values for eaeh random variable by 
sampling from eaeh variable’s statistieal distribution [38]. The variables ean be task 
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duration, cost, start and finish time which are used to compute the quantities such as the 
critical path, slack time etc. However the Monte Carlo simulations for software 
development also does not provide accurate estimates of project parameters (duration, 
finish time, cost, etc.) due to the greater uncertainties related to requirements, tools, 
resources, budget, etc. compared to many other industries [38]. This has led to the 
proposal of different approaches to project scheduling with uncertainties such as the 
application of the theory of constraints (TOC) [38] to project management which 
employs the concept of project buffers to which we frame as “Options” to accurately 
support project scheduling activities. 



Figure 13. Scheduling Process. (From: [35]) 

The project buffers or “Options” would be based on the identification, analysis 
and quantification of the risks posed by the uncertainties and in the event that sufficient 
information is not available, a "worst-case" analysis may be utilized. The amount of 
allowance provided by the buffer would be based on assessing the degree risk posed by 
the uncertainty associated with all remaining project activities. 

G. TECHNICAL PERSPECTIVE 

The second phase in the software capital investment process is heavily slanted 
towards “technical” activities and is concerned with all the technical activities needed to 
develop the software product itself The uncertainties which are typical of this phase 
include uncertainties associated with requirements analysis, transition from system 
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requirements to design and eode, uneertainties in software re-engineering and software 
reuse, and uncertainties in operating the software product itself. For example, taking the 
case of software quality attributes such as performance and maintainability, which cannot 
be directly measured until after the fact (after the software has been fielded), the activities 
associated with enhancing these quality attributes cannot be estimated or measured 
during the development process. Such concerns therefore consequently introduce 
aleatoric uncertainties in the software product. Technical uncertainties can be categorized 
into the four major activities of specification, design/implementation, validation, and 
evolution activities as identified in [1], all of which are carried out regardless of the 
acquisition strategy selected, i.e., build vs. buy, as even commercial off-the-shelf (COTS) 
products need to be tailored and customized using these activities to meet the customers 
needs. 
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Softv/are 1 

Specification 1 

Software Design &l 
Implementation 1 

Software 1 
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Software 1 
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Figure 14. Technical Activities. 

1. Software Specification 

The Software Specification phase is typically the first phase in any acquisition 
process once a software need has been identified. Requirements are elicited through 
various mechanisms and a software requirements specification is generated. The key 
underlying uncertainty facing the software specification activity is the probability of 
being faced with incomplete, unknown or changing requirements. More often than not, 
these changes are proposed once the software is already in operational use, as users begin 
to see better ways to perform existing functionality or because there is a mismatch in 
performance expectations. This scenario highlights the realities of the challenges facing 
software specification and exemplifies “Humphrey's Requirements Uncertainty 
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Principle” which posits that for any new software system, the requirements will not be 
completely known until after the users have used it. The prudent manager would ensure 
that to the extent possible, the requirements developed in this phase are testable amongst 
other things. 

2. Software Design and Implementation 

The software design and implementation phase usually follows the software 
specification phase. The tasks which are inherent to this activity are architectural design, 
detailed design and program design as depicted in the Figure 15. This phase involves the 
initial development of high-level software architecture which is continuously modified 
until the requirements are completely refined. A detailed design is generated from the 
high level architecture and the activities of coding, integration, and testing are carried out 
during this phase. Uncertainties in this phase are usually a propagation of the 
uncertainties in the software specification phase and manifest themselves in the software 
design. While recent empirical studies of software development under realistic conditions 
has lead to the introduction of several agile development process such as extreme 
Programming (XP) and SCRUM which promote development iterations, open 
collaboration with the customers, and adaptability throughout the life-cycle of the project 
in response to both Humphrey’s and Ziv’s uncertainty principles, to help address 
software engineering uncertainties, these approaches in our opinion do not adequately 
address the challenges posed in this phase as we believe agile approaches by virtue of its 
definition is somewhat more of a “reactive” approach to software development than 
proactive approach because of the elimination of what is perceived to be wasteful, yet 
necessary activities associated with traditional software development processes. Also 
while beneficial we are of the opinion that caution must be exercised when employing 
agile methods, especially in organizations that do not embrace certain standards of 
discipline that are associated with an engineering discipline (i.e. documentation, 
configuration management etc.) in order to preserve the integrity of the relatively young 
profession of software engineering as a traditional engineering discipline. Furthermore 
while modern concepts such as posteriori reuse (design for planned reuse) are 
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increasingly becoming widely advocated standards in whieh software artifaets are ereated 
with reuse in mind with the goal of improving productivity, quality, time-to-market and 
long-term eost, this concept is still evolving in the software engineering eommunity. 
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Figure 15. Software Design and Implementation Process (From: [36]). 


Uncertainties in the software design process can be presented along the 
dimensions of two major points of view; the developer’s point of view and the customer’s 
point of view. From a developer's point of view, design decisions should be evaluated as 
early as possible in the software development life cycle, and key requirements, both 
functional and non-functional requirements, must be fully addressed during the 
architectural design phase. It is not acceptable to proceed through detailed design and 
implementation phases only to diseover that the decisions made early in the process, at 
the arehiteetural level, did not turn out to promote the desired funetionality to be achieved 
[30]. 

From a customer's point of view, there are two options towards achieving their 
solution; building a component or buying the needed component. The developers’ 
concerns identified above would apply in both scenarios (build vs. buy) even in the case 
that a solution needs to be bought, beeause the eustomer is faeed with the ehallenge of 
procuring a software component that it is eapable of meeting its requirements straight out 
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of the box (which is rare) or with some customization, particularly with respect to their 
long-term needs and the system's response to change. 

While meeting current requirements is definitely a focus of both points of view, 
future evolution serves as the driving mechanism on which decision would be based, 
hence the need for a decision-making framework to mitigate theses uncertainties. 
However, before the Real Options approach can be employed, the factors which would 
guide the decision-making process or evaluation process need to be assessed. These 
factors are realized by identifying and quantifying the uncertainties that appear in 
software design. 

Software design situations are characterized by complexity and uncertainty with 
complexity representing the amount of relevant information that is available in a given 
situation and uncertainty representing the availability and reliability of the information 
that is relevant in a given situation [43]. Consequently, effective approaches to software 
design would require a closer look at complexity and uncertainty as important 
characteristics of a situation and embrace a set of simple concepts for relating situational 
characteristics to different approaches to reduce both complexity and uncertainties in 
software design. While techniques such as abstraction and decomposition are commonly 
employed to deal with the complexity associated with software design in specifications, 
approached based on prototyping have emerged as means to cope with software design 
uncertainty [44]. 

Previous research into software design approaches led to the proposition of the 
Principle of Limited Reduction [44] which concluded that software engineers should rely 
on mixed approaches to software design in order to effectively cope with both complexity 
and uncertainty because it is almost impossible to hope to reduce complexity without 
adversely increasing uncertainty and vice-versa. 

3. Software Validation 

The software validation activity serves to increase the degree of confidence in a 

software product by determining if the software product which is being delivered is the 

right software. It is normally preceded by verification activities aimed at ensuring the 
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software product is designed to deliver all functionality to the customer and typically 
involves reviews and meetings to evaluate documents, plans, code, requirements and 
specifications. 

As is the case with the software design and implementation phase, uncertainties 
which surround this activity are due to the ripple effect associated with software 
specifications and the inability to guarantee a defect free product during the testing 
process. While verification serves to verily the correctness of the software product, 
validation ensures that functionality, as defined in the requirements, is the intended 
behavior of the product and typically involves actual testing of the software product and 
takes place after verifications are completed. The validation phase serves as a quality 
assurance procedure in which processes are designed to ensure that software product 
meets its intended use and the validation activities are usually undertaken both during, as 
well as at the end of the software development life cycle to ensure that all requirements 
have been fulfilled. However due to the potential ripple effect associated with software 
specification uncertainties, this activity has the potential of being plagued by the 
“garbage-in, garbage out” syndrome. 

4. Software Evolution 

In general, the software evolution activity addresses the change associated not 
only with the software systems, but also with the case tools and the development 
processes used to support software engineering activities in order for the software system 
to remain relevant in terms of its use. In Lehman’s view, software evolution is “intrinsic” 
and not primarily due to shortsightedness of developers or users, but rather is due to the 
fact that software it is a man-made product which continuously evolves and the rate of 
evolution of a used software system will far exceed the rate of evolution of a biological or 
physical system, because the fundamental laws of biological and physical systems do not 
change, while the laws of software, being entirely man-made and malleable, can be and 
are changed at a moment’s notice [41]. 

In our view, software requirements are the driving mechanism through which 
uncertainty is introduced into the technical phase of software capital investments, with 
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uncertainties in eurrent requirements and future requirements playing a key role. In 
Lehman’s work on software evolution, he elassifies software systems into three major 
eategories; S, P and E systems based on system correetness levels whieh are eorrelated to 
the three major sourees of uncertainty whieh arise in the software engineering proeess: 
Godel-like, Heisenberg-type and Pragmatie uneertainties. These three types of 
uncertainties eould be interpreted as uneertainties in the problem domain, uneertainties in 
the solutions domain and uneertainties in human partieipation respeetively. We expand 
on these uneertainties in the next seetion. 

H. TYPES OF SOFTWARE ENGINEERING UNCERTAINTIES 

Godel-like uncertainties oecur when the properties of a program eannot be known 
from the representation, beeause the software systems and their specifieations are abstraet 
models of the real world. Heisenberg-type uneertainties oecur as the system is being 
developed and grows during use and exhibit themselves in the form of ehanging 
requirements either due to unsatisfactory behavior post implementation or beeause of the 
emergence of new requirements while pragmatie uneertainties are problems in aetually 
performing the development aetivities. 

In the ease of Godel-like uneertainties, we believe that sinee a model is only an 
abstract representation of reality and as such in most cases does not offer the level of 
detail that is desired, modeling and simulations teehniques when properly employed, 
serves as a eost saving meehanism through whieh various events/seenarios could be 
modeled in a controlled environment before aetual implementation. Furthermore this 
eould serve as a “proof of eoneept” in the form of a prototype, although even though it 
might help reduee some uneertainties, from a eost perspeetive, the uneertainty of system 
properties might still persist. 

Changing requirements are a normal oeeurrence in software development efforts. 
Therefore the ehallenge facing the software manager trying to deal effeetively with 
Heisenberg-type uneertainties is two fold: how to aeeurately manage the ehanging 
requirements and how to avoid sunk eosts as a result of an un-useable produet due to 
ineorreet or new requirements. 
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Uncertainties 

Sources of Uncertaintv' 

Softivare Development 
Process 

Employee 

(Softrvare Engineer) 

Firm 

Market Conditions 

Gbdel-like uncertainties 

Uncertainties resultiirg 
from ambiguous system 
properties 

Inadequate modeling 
techiuques and tools. 
Availability* of required 
technology. 

Lack of skilled engineers 
in the domain to help in 
ehciting system properties 

Fam/Skill profile 
mismatch with 

customer 

requirements 

Demand for new skills 

Uncertam supply of new 

skdls 

Heisenberg-type 

uncertainties 

Uncertainties resulting 
from changing 
requirements 

Lack of clear 
requnements and 
mability* to capture the 
changmg requirements. 

Ripple effects caused by 
changes to a requirements 

Lack of 

understanding of 
domain 

\*anations in demand for 
supply of services 

Pragmatic uncertainties 

Uncertainties resulting 
from perf omung 
development acthities, e g. 
cost 

IncoiTect estmiabon 
techiuques 

Erosion of existing skills 

Inability to leani ne^v 
skills 

Employee dissatisfaction, 
lack of comnutment 

High financial 
leverage 
\'ariations m 
profitability 

Business Cy'cles 

Competitive pressure for cost 
reduction 


Table 3. Summary of Software Engineering Uncertainties. 

This also introduces the challenge on how to accurately reflect or account for this 
uncertainty in the cost and schedule thereby contributing or introducing Pragmatic 
uncertainties ranging from funding problems, market conditions to employee related 
issues which could all impact the software development effort. 

In Table 3, we have attempted to highlight the three categories of the uncertainty 
and identify sources of uncertainty associated with each category. 

We use a hypothetical example to further illustrate these uncertainties in the 
scenario of developing a software application for the space shuttle. If the space shuttle is 
designed to travel to the moon and back, what are the uncertainties facing this effort and 
what kind of options would the software manager want to consider when designing this 
software? 
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Godel-like uncertainties 
Ambiguous System properties 


Modeling Options 
Investing in Modeling tools 
which model to a detailed 
level of abstraction. 


Heisenberg-type uncertainties 
Changing requirements 


◄-► 


Requirements Management Options 

Investing in Requirement Management 
tools, Making the architecture “flexible” 
enough so It can be easily evolved 


Pragmatic Uncertainties 

Determining the accurate 
cost of the software project 


Figure 16. Sample Space Shuttle “Real Options”. 


Cost Estimation Options 
Utilize a function Point Analysis method. 
Utilize the Use-Case Point Analysis Method. 
Utilize the SLOC Method, and 
Utilize the COCOMO II Methods 


A forward-thinking manager would identify future use of the space shuttle 
software as an uncertainty by thinking evolution, evolution, evolution! In this scenario, 
s/he would design the space shuttle software with evolution in mind and would attempt to 
determine the extent of evolution depending on the scenarios that he/she could come up 
with. For example, the software manager might want to make an investment in a robust 
architecture that could support a plug and play navigational system in case NASA 
decides to use the space shuttle to visit Pluto in the future so that only one component is 
changed (navigational software module). This forward thinking definitely requires some 
creativity and intuition on the part of the project manager. Figure 16 showcases these 
uncertainties and sample Real Options that can be used to manage them. 

The most likely scenarios would then be prioritized and categorized into 
uncertainties and options developed and analyzed appropriately to counter the 
uncertainties. Thus, the essence of employing a Real Options approach based to the 
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potential evolution of the software requires the enhancement of traditional requirements 
engineering processes to allow for the creative forecasting of future requirements and 
development of the appropriate options to allow for future evolution even if their might 
be technology maturation concerns. 

I. MANAGING SOFTWARE ENGINEERING UNCERTAINTIES 


In an attempt to manage software engineering uncertainties, we capture the 
uncertainties from both a managerial and technical perspective in our Uncertainty 
Elicitation Model and visually depict the contributing factors in Figures 17 and 18 in 
what we call the “2 T’s” of software engineering uncertainty. (Given that our depiction 
resembles the capital letter “T”.) 
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Figure 17. Software Engineering Management Uncertainties. Managerial 
Uncertainties of people, time, functionality, budget, and resources 
contribute to both estimation and schedule uncertainties which are 
considered to be pragmatic uncertainties. 
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Figure 18. Software Engineering Technical Uncertainties. Technical 
uncertainties of incomplete requirements, ambitious, ambiguous, 
changing/unstable requirements contribute to software specification 
uncertainties, which leads to software design and implementation, 
software validation and software evolution uncertainties all of which 
can be categorized as exhibiting both Heisenberg- type and Godel-like 

uncertainties. 


The “2T” approach would serve as the basis on which uncertainties are elicited 
during the elicitation phase as proposed in our Uncertainty Elicitation Model. An 
expanded view of the model is depicted in Figure 19. 
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Figure 19. Expanded View of Uncertainty Elicitation Model. 

Given all the uneertainties that appear in software engineering, one must try to 

reduee and/or eliminate the uneertainties to enable a more aeeurate end result in the 

software investment process. Hence, we need a quantification mechanism to quantify 

these uncertainties as risks with the goal being to assign an appropriate mathematical 

model to a real-world situation with respect to objective and subjective uncertainty [32]. 

While there are several mathematical theories that describe uncertainty and provide its 

measures such as probability, possibility and evidence theory, probability theory and 

possibility theory are best suited for describing aleatoric uncertainty and epistemic 

uncertainty respectively. Evidence theory, on the other hand, is well suited to handle both 

types of uncertainty because it does not require the separation of the two types of 

uncertainties due to its unique ability to represent the degree of belief (confidence) that 

may be attributed to a given proposition on the basis of given evidence, and its ability to 

combine evidence from different sources (Dempster’s rule) [31]. Consequently, as the 

next step in our analysis we propose characterizing both the management and technical 
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uncertainties as depicted in Figure 20 by modeling the uneertainties within a single model 
developed based on the logic of evidence theory so as to further delineate between 
aleotoric and epistemic uncertainties, determine the uncertainties that could be reduced, 
understand the dependenee between events and act accordingly by developing Real 
Options to mitigate the risks perpetrated by the uncertainties. 


People 

Time 





Figure 20. Modeling Software Engineering Uncertainties. 

Both the Managerial and Teehnical uncertainties are fed into a risk model and epistemic 
and aleotoric uncertainties characterized from the inputs 


Up to this point, we have suceeeded in proposing a methodology for eliciting 
both the managerial and technical uncertainties surrounding a software-related capital 
investment. Onee we have eaptured all the possible uneertainties, we determine the risk 
which these uncertainties present and characterize the risk as a measure of volatility on 
the software investment effort so that the appropriate options can be developed to hedge 
against the risk. 

From a Real Options perspective, uncertainty is the randomness of outcomes from 

a software investment deeision. Real Options are implieit or explicit capabilities created 

for “real assets” that provide the software manager with time-deferred and flexible 
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choices (options) regarding future ehanges of the software [28]. Real Options theory 
explieitly addresses the eoneems of software investment ehoiees for future eapabilities. It 
assumes that managers develop a level of foresight sufficient to invest in 
resourees/teehniques and proeesses with ‘options’ eharacteristies that provide implicit or 
explicit claims on future opportunities and generate flexibilities for future investments or 
ehanges [28]. Through these eapabilities, the software manager may ehoose to adjust, 
reduee, inerease, or abandon the investment in the future, thereby stabilizing returns from 
these assets. 
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IV. ESTIMATING VOLATILITY 


A. INTRODUCTION 

In the previous ehapter we addressed the issue of uncertainty and follow up in this 
chapter by introducing a methodology for estimating volatility as we continue to develop 
our Real Options (RO) framework. As previously discussed in this study, uncertainty 
implies risk, and consequently, uncertainty must be duly quantified as a risk factor to 
gauge the magnitude of its impact on the underlying asset. The process of translating or 
equating software engineering uncertainties into a quantifiable property begins with the 
quantification of the identified uncertainties, computing the impact of uncertainties and 
ultimately developing a risk analysis framework in which the associated risks are 
identified, predicted and modeled using simulation and the results analyzed. This 
approach is representative of the current approach to risk management. However in our 
study, we seek to proceed beyond the conventional paradigm of risk analysis by explicitly 
utilizing Dempster-Shafer Theory of Evidence (DST) to reduce both aleotoric and 
epistemic uncertainties by using subjective probability estimates to refine our objective 
estimate thereby further acknowledging the flexibility of software manager to make 
decisions to incorporate Real Options to mitigate and hedge against risk. 

We specifically choose to use the DST approach because it accounts for 
ignorance, a key strength and value added capability to our analysis since software 
engineering is unique in that there are several different software development approaches 
and processes without any standardization across the industry. Therefore it is our belief 
that DST when applied, could address the issue of ignorance or “unknown unknowns” in 
our probability estimates which otherwise reflect “known unknowns” such as in 
situations where it is not immediately clear from the historical data as to the type of 
software development approach that was utilized in sample historical projects (e.g.. 
Rational Unified Process, Evolutionary Approach, Component Based Software 
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Engineering). Our subjeetive probability estimate based on DST would help establish a 
eonfidenee interval on the key objective probability estimates to consequently reduce 
uncertainty. 

B. RISK OVERVIEW 

From a Real Options perspective, estimating volatility as a risk measure 
represents the most challenging task associated with the application of Real Options. 
Software risk can be broadly defined as a measure of the likelihood and loss of an 
unsatisfactory outcome affecting the software project, process and product. Risk can be 
captured in different dimensions and it could be estimated in terms of the volatility of the 
returns on the underlying asset or the degree of uncertainty about the future value of the 
opportunity. In general the risks associated with software-related capital investments 
can be analyzed in one of two possible ways: an objective approach, which is quantitative 
in nature in which risks are analyzed by calculating the probability of their occurrence 
and impact on the investment effort under consideration and a subjective approach which 
is qualitative in nature and is based on an experts opinion, intuition, assessment based on 
degrees of possibility about the risks rather than the reliance of historical experiences. 

In general, there are nine major theories in decision making that guide risk 
management. These theories are Bayes theorem. Chaos theory. Creativity theory. 
Decision theory. Game theory. Portfolio theory. Probability theory. Uncertainty theory 
and Utility theory. Bayes theorem addresses the dynamic nature of risk by providing a 
method to alter judgment as events unfold. Chaos theory posits that chaos and 
uncertainties are market opportunities which must exploited. Creativity theory allows us 
to use our knowledge and imagination to develop ideas that are either original or novel. 
Decision theory uses probabilities to determine outcomes in complex problems. Game 
theory uses heuristics to determine which alternatives to explore in large search spaces. 
Portfolio theory utilizes the concept of diversification to reduce risk. Probability theory 
uses probability estimates to determine the degrees of certainty and forecast an outcome. 
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Uncertainty theory uses probability to model unknown, uncertain or subjective deeision 
problems, and finally Utility theory, whieh models preferenees towards risk by seleeting 
the alternatives that maximizes the expeeted utility funetion [45]. 

Evidenee theory on the other hand, otherwise known as Dempster-Shafer Theory 
of Evidenee, whieh we utilize in our methodology, originated from Bayes Theorem of 
probability analysis. It has its strengths in its ability to represent and eombine different 
types of evidenee obtained from multiple sourees. However, it requires the satisfaetion of 
two eonstraints, the first being the independenee of the sourees of information and 
secondly the handling of eonflieting evidenee. We will expand on our methodology using 
DST later on in this chapter, but first we revisit the issues of risk estimation. 

1. Software Investments Risks Estimation 

Based on the uneertainties we identified using our “2T” approaeh proposed in our 
Uneertainty Elieitation Model (Eigure 19) in the previous ehapter, we now attempt to 
determine and quantify the risks posed by these uneertainties. We examine the risk of 
requirements ehanges (teehnieal aetivity), eost and sehedule overruns (managerial 
activity) based on the uncertainties identified as well as the impaet these risks pose to the 
relative future value of the software investment effort. We use the term “relative future 
value” beeause the future value of government investments is not easily quantifiable. We 
further expand on this in the next chapter. These risks are then modeled using a Monte 
Carlo simulation to determine the volatility of the software investment effort. 

2. Requirements Risk Estimation 

Based on our researeh thus far we ean safely assert that both Godel-like 
uneertainties and Heisenberg-type uneertainties assoeiated with software related eapital 
investments are key drivers of pragmatie uneertainties. We put emphasis on the word 
“key” beeause there are other external eontributing faetors that also eontribute to 
pragmatie uneertainties. However, Heisenberg-type uneertainties, speeifieally 
requirements issues serve as the key initiator of the uncertainties assoeiated with software 
related eapital investments and requirements volatility and inadequate requirements have 
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a significant impact on either the suecess or failure of the software investment effort. 
Typieal ehanges might involve ehanges in funetional, level of serviee (interoperability, 
performanee), design eonstraints, quality attributes, and interfaee requirements amongst 
others. Furthermore, teehnologieal breakthroughs or unantieipated advancements in the 
software industry while development is underway or future evolution of the software 
produet eould also eontribute to requirements ehanges. 

Ever ehanging requirements eontinues to impaet software investment efforts, and 
more often than not, it forees managers to ehoose between requirements, i.e., whieh 
requirements to accept and which requirements to reject with the full understanding that 
ignoring ehanges in requirements has the eonsequenee of the delivered product failing to 
meet the customers needs while accepting ehanges in requirements has the potential of 
impaeting eosts and sehedule. Furthermore, ehanges in requirements while a software 
investment effort is underway also poses the risk of introdueing unwanted, unantieipated 
or unknown impaet on existing requirements, not to mention the assoeiated eosts and 
seheduled delays depending on the phase of the investment or software development 
proeess experieneing signifieant requirements ehanges. While the standard practice has 
been to “freeze” requirements prior to the eommeneement of any development aetivities, 
more often than not, this does not work and is also not representational of the DoD 
doetrine to support the flexible development and rapid delivery of products to meet the 
war-fighters needs in an ever changing environment in response to operational needs. 

Therefore in order to aeeurately estimate requirements volatility and its impaet on 
the future value of a software-related eapital investment, not only must the risk of 
requirements ehanges be quantified, it must also be speeifieally predieted and quantified 
based on the phase in the software development proeess in whieh the ehanges are more 
likely to oecur. Hence the need of an approach that would explicitly acknowledge not 
only the probability of oeeurrence based on previous objeetive estimates, but also the 
possibility of oeeurrenee based on subjeet expert opinions (Delphi Method) that 
aeknowledges either the degree of belief or ignoranee in the objective probability 
estimates whieh we attempt to address using the DST approaeh. 
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3. 


Schedule Risk Elicitation 


In general software engineering scheduling risks could be addressed either from a 
development standpoint or delivery standpoint, although the development schedule 
ultimately drives the delivery schedule. While the decision variables which typical drive 
the scheduling activity could be either managerial or technical in nature, we posit that 
technical issues present the most risk because managerial risks could easily be controlled. 
Specifically the decision variable of requirements changes in the form of requirements 
increases or reductions has the unintended consequence of impacting the software 
delivery schedule by either pushing the delivery date out by months and in some case 
years past the anticipated delivery date or shortening the delivery schedule in response to 
either operational needs, cost constraints, or as a result of risk mitigation. This dilemma is 
further highlighted in Figure 21 below which showcases the difference between the 
anticipated delivery date of a product and the actual delivery date of a software product 
as the size of the software product increased due to requirements changes. 



-Planned 

-Actual 


Figure 21. Differences Between Planned and Actual Software Schedules 

(From: [38]). 


Therefore in order to hedge against the uncertainty of requirements changes, 
better and flexible estimation techniques which account for the changes in requirements 
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need to be developed, although schedule estimation is beyond the scope of our study. In 
our study, we assume the schedule estimation techniques are sufficient enough for us to 
provide initial schedule estimates which we can then refine, by addressing the uncertainty 
that was factored into the cost estimation using our DST methodology. 

4. Cost Estimation Risk Elicitation 

Initial estimation of the costs of the software investment effort is a critical 
component of the software investment process as it provides a basis for justifying the 
investment effort. This estimate must not only account for all the normal tasks and 
activities which needs to be accomplished during the whole investment effort, but must 
take into consideration other factors to include requirements volatility, time for resolving 
defects time associated with “spin up” during contract re-competes and possible litigation 
if several contractors are involved as is the case with several DoD acquisition programs 
to name a few. Therefore it is paramount that all the pertinent decision variables that 
impact the overall cost of the software investment effort be accurately captured. 
However, given the three sample decision variables of requirements, schedule and cost 
changes, the question now becomes how do we capture this information? For this we now 
resort to the next phase in our methodology which is the evidence gathering phase. 

C. EVIDENCE GATHERING 

In order to estimate the volatility of the returns associated with our current 
software investment effort, we need to gather evidence to help derive our estimates. 
Historically, gathering of evidence using previously completed software-related capital 
investments as a proxy is a difficult task for the following reasons: 

1. The current software investment effort under consideration might be the 
first of its kind with no known comparables 

2. Information is rarely or actively collected and managed in a disciplined 
fashion 

3. Even when information is collected, accessibility by third parties is 
usually difficult due to the proprietary nature of the information. 
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Thus more often than not the software executive is faced with the identifying 
alternate sources of information to either assert or dispute their initial volatility estimates. 
In our study, we propose the use of two sources of information, the first being historical 
data (objective approach) and the second being expert opinions obtained using the Delphi 
method (subjective approach). We choose to use both methods because we believe 
intuition and judgment (subjective approach) should supplement quantitative analysis 
(objective approach). More often than not, past success and failures serve as key 
indicators of the future. Thus historical data can be used to predict and explore “what-if ’ 
scenarios on future projects based on the use of forecasting and analytical analysis. 

The Delphi Method is a technique first introduced by the RAND Corporation in 
the 1940’s as a methodology for the elicitation of the opinion of an expert or groups of 
experts to guide decision making by the making predictions about future events. It places 
emphasis on a iterative, systematic, disciplined and interactive process of individual 
interviews (usually conducted using questionnaires) and the outcome is based on the 
Hegelian Principle of achieving consensus through a three step process of thesis, 
antithesis, and synthesis [48]. In the thesis and antithesis steps, the team of experts 
present their opinion or views on the given subject, establishing views and opposing 
views, and consensus is ultimately reached during the synthesis phase as opposing views 
are brought together to form the new thesis. Widely used as an estimating tool, the Delphi 
Method has been used to estimate values for factors which appear in software estimation 
models such as cost estimation [47] and risk estimation. Furthermore, it is one of the 
approved techniques published by the U.S. Army Cost and Economic Analysis Center in 
February 2001 for preparing or reviewing economic analyses in support of the decision 
making process. 

In the event that there is no historical data available, we resort to obtaining 
information using the Delphi Method based on a minimum of two independent estimates 
provided by two independent experts or two separate groups of independent experts. The 
first estimate would provide the base case estimate with the remaining two independent 
estimates serving to asses or refine the initial estimate independently. In the case that we 
do have historical data but are unable to find projects meeting any or all of the criteria 
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above, we proeeed to “fit” the data to as elose as possible to mimic our current software 
investment effort by employing interpolation techniques to understand and forecast our 
project based on the trends depicted in the historical data. 

1. Data Fitting Techniques 

To fit our data, we proceed to determine if the risks depicted in the historical data 
are positively correlated to the risk presented in our current project. We proceed to 
conduct a regression analysis by modeling and analyzing the historical data consisting of 
values of the dependent variable as a function of one or more independent variables and 
an error term which represents unexplained variation in the dependent variable. Each of 
the parameters is estimated so as to give a "best fit" of the data using the least squares 
method. The least squares method corresponds to the maximum likelihood criterion if the 
experimental errors have a normal distribution and the best fit in the least-squares sense is 
that instance of the model for which the sum of squared residuals has its least value, a 
residual being the difference between an observed value and the value given by the model 
[69]. 

The least squares problems fall into two categories, linear and non-linear 
representative of linear and non-linear data in our case. The linear least squares problem 
has a closed form solution, but the non-linear problem does not and is usually solved by 
iterative refinement; at each iteration the system is approximated by a linear one, so the 
core calculation is similar in both cases [69]. 

Confidence limits can be found if the probability distribution of the parameters is 
known, or an asymptotic approximation is made, or assumed. Likewise statistical tests on 
the residuals can be made if the probability distribution of the residuals is known or 
assumed. The probability distribution of any linear combination of the dependent 
variables can be derived if the probability distribution of experimental errors is known or 
assumed. However running a regression analysis introduces two problems; 
autocorrelation and multicollinearity. 

Multicollinearity occurs when two or more explanatory variables in the regression 

model are highly correlated, making it difficult or impossible to isolate their individual 
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effects on the dependent variable [68]. It is easily solved by obtaining more data or 
dropping one of the variables. Autocorrelation on the other hand occurs when the error 
term in one time period is positively correlated with the error term in the previous time 
period (positive first-order). This leads to downward-biased standard errors (and, thus, to 
incorrect statistical tests and confidence intervals). The presence of autocorrelation can be 
tested by calculating the Durbin-Watson statistic below [70]. 

y(*# "•*-!)* 

- 



Its value always lies between 0 and 4. A value of 2 indicates there appears to be 
no autocorrelation [70]. If the Durbin-Watson statistic is substantially less than 2, there is 
evidence of positive serial correlation. As a rough rule of thumb, if Durbin-Watson is 
less than 1.0, there may be cause for alarm [70]. Small values of d indicate successive 
error terms are, on average, close in value to one another, or positively correlated. Large 
values of d indicate successive error terms are, on average, much different in value to one 
another, or negatively correlated [70]. 

Once we have completed the evidence gathering phase, we are now ready to 
proceed to the analysis phase in which we analyze the information to obtain the volatility 
of our software investment effort. To accomplish this, we utilize a Monte Carlo 
Simulation approach. 

D. VOLATILITY ESTIMATION: MONTE CARLO SIMULATION 

Volatility is used to quantify the risk of the asset based on the standard deviation 
of the changes of the decision variable within a specific time period divided by its mean 
to obtain a common sized percentage volatility measure (and the result is annualized). In 
conventional financial investment theory, the Capital Asset Pricing Model (CAPM) is the 
most commonly used technique for assessing market risk. The CAPM formula takes into 
account the asset's sensitivity to non-diversifiable risk (also known as systemic risk or 
market risk), often represented by the quantity beta (P) in the financial industry, as well 
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as the expected return of the market and the expected return of a theoretical risk-free 
asset. From a Real Options perspective, risk is a key driver in the value of a real option, 
and is positively related to value. In the traditional NPV approach, high volatility 
signifies high risk, high risk signifies high discount rates, and high discount rates mean 
lower values. However, a high volatility is linked to high value in Real Options valuation 
because greater volatility creates a wider range of possible future values of our 
opportunity as we only will exercise our option if the value of the opportunity exceeds 
the exercise price [46]. Greater uncertainty on the down side would not hurt as we simply 
will not exercise. However, greater uncertainty on the up side produces a greater excess 
of opportunity value over exercise price, and a correspondingly greater option value [46]. 

A Monte Carlo simulation which is the simulation technique we employ in our 
study is an advanced computer-based simulation technique which is more or less an 
extension of traditional simulation techniques of sensitivity and scenario testing in which 
the critical success drivers or variables that affect the bottom-line variables are simulated 
thousands of times taking into account interdependencies between the variables to 
emulate all potential combinations and permutations of outcomes [14]. We therefore 
model the risks facing our software investment effort using a Monte Carlo approach, and 
specifically employ the Risk Simulator software provided by Real Options Valuation, 
Inc. to derive the volatility of the software investment effort. 

Estimating volatility from historical data or from the “fitted data” obtained from 
extrapolating historical data is a fairly simple process. We simply apply the logarithmic 
relative returns function and then take the sample standard deviation on it, annualize it by 
multiplying by the square root of periods in a year. However, in the case of the evidence 
obtained using the Delphi Method, we cannot directly apply this approach because the 
information elicited is based on subjective probabilities, which cannot be modeled in the 
traditional sense that classical objective probabilities are modeled. As mentioned earlier 
on in the chapter, this volatility estimate gives an objective assessment based on the 
known unknowns and in order to account for the software executive ignorance in the case 
of “unknown unknowns”, especially in innovative software investment efforts, we 
employ DST to further refine our objective estimates. In other words, the volatility 
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estimate eomputed at this point using the historieal data might have little or no 
relationship with the potential volatility of our eurrent software investment effort, 
beeause other than telling us that similar projeets experieneed a eertain amount of 
volatility, it does not tell us the volatility of our software investment effort and the 
historieal estimates might not neeessarily refleet intrinsie details that refleet the 
differenee between our eurrent projeet and the historieal projeets. 

Therefore to refine the historieal volatility estimate of the previously eompleted 
software investment, we propose the solieitation of additional information using several 
expert opinions based on the Delphi Method (at least 2 opinions) as our souree of 
information and then use the DST approaeh to eombine the information we reeeived from 
the experts and establish boundaries of the historieal volatility estimates obtained in the 
form of beliefs and plausibility based on subjeetive probabilities that take into 
eonsideration unique attributes of the eurrent software investment effort. We further 
explain the meehanies of the DST approaeh in the next seetion. 

E. VOLATILITY REFINEMENT USING DEMPSTER SHAFER’S THEORY 

As mentioned in our introduetion, DST is well positioned to address both 
epistemie and aleotorie uneertainty. While traditionally, probability theory has been used 
to eharaeterize both types of uneertainty, reeent eritieisms of the probabilistie 
eharaeterization of uneertainty elaim that traditional probability theory is not eapable of 
eapturing epistemie uneertainty [50]. DST is a mathematieal theory of evidenee originally 
developed by Dempster in 1967 [49] with roots embedded in Bayesian statistieal analysis 
and later expanded by Shafer in 1976 [49]. While Bayesian inferenee requires all 
unknowns to be represented by probability distributions, whieh awkwardly implies the 
probability of an event for whieh we are eompletely ignorant, DST takes over by 
introdueing belief funetions to distinguish ignoranee and randomness by assigning 
probability mass to subsets of parameter spaee, so that randomness is represented by the 
probability distribution and uneertainty is represented by large subsets [72]. In other 
words while Bayesian theory requires probabilities for eaeh uneertainty of interest, the 
theory of belief funetions provides a non-Bayesian way of using mathematieal probability 
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to quantify subjective judgments [51]. It measures degrees of belief [or confidence] for 
one uncertainty on the probabilities for a related uncertainty. 

The premise behind DST is it can be interpreted as a generalization of probability 
theory where probabilities are assigned to sets as opposed to mutually exclusive 
singletons. In the case that there is sufficient evidence to permit the assignment of 
probabilities to single events, the Dempster-Shafer model collapses to the traditional 
probabilistic formulation where evidence is associated with only one possible event [50]. 
DST relies on three basic functions: the basic probability assignment function, a primitive 
of evidence theory which does not refer to probability in the classical sense, and two non¬ 
additive continuous measures called Belief and Plausibility which are both used to 
combine separate pieces of information (evidence) to calculate the probability of an 
event, while at the same time defining the upper and lower bounds respectively of an 
interval that contains the precise probability of a set of interest [23]. 

Since evidence can be associated with multiple possible events, e.g., sets of 
events, the evidence in DST can be meaningful at a higher level of abstraction, a key 
benefit needed at the strategic decision-making level without having to resort to 
assumptions about the events within the evidential set [50]. Furthermore the DST model 
can be used to cope with varying levels of precision regarding information with no 
further assumptions needed to represent the information as demonstrated during a study 
in addressing uncertainties in systems [50]. We posit that the demonstrated approach also 
allows for the direct representation of uncertainties associated with software-related 
capital investments since we can characterize vague inputs as sets or intervals with the 
resulting output also being a set or an interval. 

1. Mechanics of Dempster-Shafer Theory 

DST is a theory about two things: 1) Degrees of belief and 2) Weights of 
evidence, with a key benefit of DST being the ability to represent ignorance in the face of 
uncertainty especially when there is no information so far. In probability theory, uniform 
distributions are used to represent ignorance, however the problem with this approach is 
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that we represent the spaee of possibilities affected by the probabilities we get [71]. The 
theory of belief functions is based on two ideas [51]; 

1) The idea of obtaining degrees of belief for one question from subjective 
probabilities of a related question, 

2) Dempster's rule for combining such degrees of belief when they are based on 
independent items of evidence. Degrees of belief obtained in this way differ from 
probabilities in that they may fail to add to 100%. 

Both ideas are consistent with the Real Options pre-conditions as the degrees of 
belief are established on a frame of discernment meant to address uncertainty. DST starts 
off by assuming a Universe of Discourse 0 otherwise known as the Frame of 
Discernment which is a set of mutually exclusive alternatives. 

Thus a frame of discernment A of a set of mutually exclusive alternatives or 
possibilities can be represented as 

0={Ai .A„}. (1) 

where Ai through An represents the set of possibilities or mutually exclusive alternatives. 
A key stipulation of DST is that it should be only used to combine belief functions that 
represent independent items of evidence. The independence required is simply 
probabilistic independence applied to the questions for which we have probabilities, 
rather than directly to the question of interest. In other words it means that the sources of 
information (or at least their current properties as sources of information) are selected 
independently from well-defined populations. For example, given the situation of 
determining if a specific software requirement would increase program costs (0i) or not 
increase program costs ( 02 ), we can represent the number of possibilities as 

0 = { 0 ., 02 }.( 2 ) 

We can then develop a set of propositions based on the three axiomatic propositions of 
belief functions [71]. These axioms are: 

{B\)Bel (0) = O. (3) 
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(B2) Bel ( 0 ) = 1 


(4) 


(B3) Given a set of subsets of 0 , Ai.An 

Bel (Ai U.U An) > S Bel (A.) - S Bel (A, f) Aj) + ...+ {-If^^Bel (Ai f).fl An).... (5) 

where axioms (Bl) and (B2) are assertions from probability theory asserting neeessary 
things are maximally likely while impossible things are minimally likely, and axiom (B3) 
asserts the sub-additive nature of DST, whieh is a key departure from the probability 
rules. Speeifieally this is highlighted in inelusion-exelusion rule for probabilities with the 
“equals” (=) symbol being replaeed by the “greater than equals to” ( >) symbol. If there 
is no evidence to support either hypothesis 0 1 or 0 2 (to show that the software 
requirement would increase or not increase program costs), then a belief function can be 
developed such that: 

5e/({0i}) = O = 5e/({0 2}). (6) 

where “0” corresponds to having no evidence, a key input assumption of DST, which is a 
deviation from the school of thought in probability theory in that probability theory 
stipulates the sum of all probabilities must equal 1 as depicted in the equation 7 below. 


P({0i}) + P({02}) = P(0)=l . (7) 

Thus under DST the following claim can be asserted: 

5e/({0i}) + 5e/({0 2}) < 1 . (8) 

and 

Bel ( 0 ) = 1 and Bel (A) = 0 for every A ci 0 . (9) 


where the belief function (Eqn 9) represents ignorance due to the lack of information to 
support either hypothesis (0 1 or 0 2 ) regardless of the set of possibilities in 0 and 
asserts that some possibility in 0 must be true hence both possible propositions are 
equally likely. Furthermore, since Bel{{@i}) and Bel{{& 2 }) are independent (i.e. 
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historical data and information provided by the panel of experts - the Delphi Method), 
the following elaim ean be made: 

= 1/3, Bel{{& 2 }) = 0. (10) 

Eqn 10 holds because the evidenee supporting one hypothesis is not evidence 
against its rivals until a fair bit of evidenee has been gathered. In other words, it is not a 
zero-sum game until late in the game [71]. In the ease that we have “Masses” of 
probabilities, belief in a hypothesis is eonstituted by the sum of the masses of all sets 
enelosed by it (i.e. the sum of the masses of all subsets of the hypothesis). Masses 
represent probabilities that are strietly additive, however they are distributed over all 
propositions rather than the singletons of 0 and they eneode degrees of belief [71]. 
Therefore a belief funetion based on masses ean be represented as a mass function, whieh 
is basieally a funetion on the subset of possibilities 2© satisfying the two axioms below 
(Eqn 11 & 12), leading to the derivation of a function m: 2©-^ [0,1] called a basic 
probability assignment. 


(Ml) = m(0) = O. (11) 

(M2) ^ . (12) 

where 


m(A) is A’s basic probability number whose value expresses the proportion of all 
relevant and available evidence that supports the claim that a particular element of 0 (the 
universal set) belongs to the set A but to no particular subset of A [50]. In other words it 
represents the strength of some evidenee or our exact belief in the proposition represented 
by A. The value of m(A) pertains only to the set A and makes no additional elaims about 
any subsets of A. Any further evidenee on the subsets of A would be represented by 
another basie probability number, e.g., B c A. m(B) would the basie probability number 
for the subset B [50]. The funetion m: 2©-^ [0,1] beeomes a belief function if it satisfies 
Bel {(d) = 0, Bel (0) = 1. (13) 

Sinee masses represent probabilities that are strictly additive, and eneode degrees 
of belief accordingly, the lower bound belief for a set A defined as the sum of all the 
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basic probability assignments of the proper subsets (B) of the set of interest (A) (Be A). 
In other words, it is the degree of belief in A which is the sum of all the mass allotted to 
subsets of A and ean be represented by the following formula [71]. 

Bel{A)= ^m(B). (14) 

BcA 

Plausibility on the other hand is 1 minus the sum of the masses of all sets whose 
interseetion with the hypothesis is empty. It is an upper bound on the possibility that the 
hypothesis eould possibly happen, i.e., it "could possibly happen" up to that value, 
beeause there is only so mueh evidenee that eontradiets that hypothesis [23]. In essenee 
plausibility expresses how mueh we should believe in A if all eurrently unknown faets 
were to support A. Plausibility ean be represented as followsi 

Pl{A) = \-Bel(K) . (15) 

where 

A is the complement of A in the classical sense. Consequently given the computation of 
both belief and plausibility above, the preeise probability of an event in the classical 
sense lies between the upper and lower bounds established by plausibility and belief 
respectively. In the event that plausibility equals to belief as shown in Eqn 15, all the 
probabilities are elassical probabilities 

Bel(A) = P(Aj = Pl(A) . (16) 

F. DEMPSTER’S RULES FOR COMBINATION OF EVIDENCE 

Combining information or evidenee from multiple sourees (historical data and 
Delphi method) in the form of belief assignments serves to aggregate the information 
with respect to its constituent parts. Dempster proposed a standard combination rule 
which can be represented as [50]; 

mn{A)= Y. when A ^ (d . (17) 

BnC=A \ — K 

where 
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K= ^ml(B)m2(C) 

BoC=0 

which is computed by summing the products of the belief probability assignments (bpa’s) 
of all sets where the interseetion is null represents basie probability mass assoeiated with 
conflict, and mi 2 (A) is calculated from the aggregation of two bpa’s m\ and m 2 . 

1. Mechanics of Dempster Rule of Combination 

Assuming we have two pieees of evidenee, based on say historieal data and expert 
judgment (Delphi Method), we now eombine the pieees of evidenee from both sourees 
using the Dempster’s eombination rules by eomputing the orthogonal sum of both pieees 
of evidenee. First we determine all the pairs of sets whose interseetion is A for a given set 
A sueh that 

Ai nA 2 = A. We then add up the basie probability assignments mi(Ai) and 
giving us 


Xm,(A,)m,(A,). {19) 

Aj nA2 =A 

Then the orthogonal sum of m\ and m 2 defined by m = mi © m 2 eould then be 
given as m(0) = 0 and is demonstrated in the matrix below (Table 4) in whieh we 
eompute the orthogonal sum of three hypothetieal risk faetors (Riskl, Risk2 and Risk} 
affeeting a software investment program based on two independent expert assessments. 




Independent Eipert 1 Subjective estimates on Objective probabilities 


Risk Factors 

|Riskl|m1=0.30 

|Riskl,Risk2|ml=0.15 

|Risfcl,Risk2,Rist3| ml = 0.05 

Independent Eipeit 2 

|Risk1| 

m2 =0.70 

m1(|Risk1|)'m2(|Riskl|j 

ml||Riskl.Risk2||'m2||Riskl|| 

m1(|Risk1.Risk2.Risk3|j'm2(|Riskl|| 

Subjective estimates 
on 3 Objective 

|Risk1.Risk2| 

m2 =0.20 

ml||Riskl||'m2||Riskl.Risk2|| 

m1||Riskl.Risk2|j‘m2(|Risk1.Risk2|) 

m1(|Risk1.Risk2.Risk3||'m2(|Riskl.Risk2| 

probabilities 

|Risk1.Risk2.Risk3| 

m2 =0.10 

ml||Riskl||'m2||Riskl.Risk2.Risk3|) 

m1||Riskl.Risk2|)'m2||Risk1.Risk2.Risk3|| 

ml||Riskl.Risk2.Risk3|)'m2||Riskl.Risk2.Risk3) 


Table 4. Orthogonal Sum Of Basic Probability Assignments. 


71 










Based on the matrix (Table 4), we ean obtain the resulting three evidenee funetions. 


«^i©«^2({Riskl}) 

= mi({Riskl})*m 2 ({Riskl)} + mi({Riskl,Risk2})*m2({Riskl}) 

+ mi({Riskl,Risk2,Risk3})*m2({Riskl}) + mi({Riskl})*m 2 ({Riskl, Risk2}) + 
mi({Riskl})*m 2 ({Riskl, Risk2, Risk3}) 

mi © m2({Riskl,Risk2}) 

= mi( {Riskl ,Risk2})*m2( {Riskl ,Risk2}) 

+ mi({Riskl, Risk2})*m2((Riskl, Risk2, Risk3}) 

+ mi({Riskl, Risk2, Risk 3})*m2({Riskl, Risk2}) 

mi © m2({Riskl,Risk2,Risk3}) 

= mi( (Riskl ,Risk2,Risk3 })*m 2 ( (Riskl ,Risk2,Risk3 }) 

We now proeeed to put all these eoneepts together through the use of an example in the 
following seetion. 

G. VOLATILITY COMPUTATION: EXAMPLE 

In order to estimate the volatility of the future returns on a software investment 
effort based on requirements, seheduling and eost estimation risks using our approaeh, we 
develop a simple hypothetieal example in whieh a software exeeutive needs to make an 
investment in the software platform of a KC-X Air refueling aireraft. In this example, the 
key assumptions are as follows: 

The diseounted future benefits or value of the investment program ($305 million) 
has been justified using traditional Net Present Values (NPV) analysis with a projeeted 
eost of ($55 million) and resulting in a NPV of $250 million (we diseuss the topie of 
valuing software investments in the next ehapter). Similarly based on a preliminary 
analysis of the software investment program using the “2T” approaeh proposed in the 
previous ehapter, the exeeutive has identified requirements uneertainties as being a key 
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contributing factor to risk. We further assume there is historieal data available and we are 
able to eome up with initial estimates based on the historieal data at hand. The exeeutive 
also has the option of using only estimates obtained from the Delphi Method (expert 
opinions). However, the exeeutive deeides to use historieal data. The projeet was 
projeeted to eontain 1000 requirements based on the Use Cases and expeeted to be 
delivered in 12 months. Therefore based on these underlying assumptions, we develop 
the framework of this problem in a model and run a Monte Carlo simulation using the 
Risk Simulator software provided by Real Options Valuation, Ine. 

Speeifieally, the software exeeutive starts of by developing and fitting his data 
estimates based on historieal data of previous programs of similar size, seope and 
eomplexity and ereates three seenarios. A, B and C, of the most likely, least likely and 
likely based on previous experienees depieted in the historieal data. Next we seleet a 
probability distribution. Sinee we are dealing with random quantities whieh are positive 
in nature and whose values eannot fall below zero we seleet the lognormal distribution 
whieh utilizes the following four mathematieal eonstruets to eompute four key measures 
known as “moments” (returns, risk, skewness and kurtosis) of our logarithm distribution 
(A moment is used to deseribe the eentroid of a statistieal set of data eneapsulating the 
area of eoneem). These measures are eomputed using the following formulas as outlined 
in [14] 

[ln(x) - ln(//)]^ 

f(x) = — ^ e 2[ln(cr)]2 x > 0; // > 0, cr > 0 

x-yJlTT ln( cr) 


mean = exp 


V ^ J 


standard deviation = -^exp(cr^ + 2//)[exp(cr^)-1] 


skewness = 


^exp(cr^)-l '2 + exp(cr^)) 


exeess kurtosis = exp (4cr^) + 2 exp(3cr^) + 3 exp(2cr^) - 6 


73 



The model is run and the simulation calculates numerous scenarios of the model 
by repeatedly picking values from the lognormal distribution for the uncertain variables 
using the values in the model. Based on the model results, the volatility of the software 
investment program due to risk of requirements increase was computed by the model as 
0.0017. This volatility measure was determined by computing the logarithmic returns of 
our $305 million investment based on the risk of requirement changes over the three 
scenarios. This volatility resulted in the cost exceeding the projected budget by 
$1,829,697.42 and consequently led to the reduction of NPV from $250,000,000 to 
$248,170,302.58. 



_ ^ 

Type Two-Tait Lower; 4nfinity. Upper; Infinity. Certainty; 100.0000% 

Figure 22. NPV Returns on $305 million Software Investment. The frequency 
histogram shows the frequency counts of values occurring in the total 
number of trials simulated. The vertical bars shows the frequency of a 
particular NPV occurring out of the total number of trials, while the 
cumulative frequency depicted by the smooth line shows the total 
probability of all values at and below the maximum NPV occurring in 

the forecast. 


Figure 22 above depicts a graphical distribution of the returns associated with our 
investment while Figure 23 on the next page depicts the statistics associated with the 
computations. 

In further analysis conducted in the model to determine the likelihood/probability 
of requirements changes by phase in the software development process, it was 
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determined that the probability of requirements increases during the specification, 
development and validation phase of the software development process were 0.32, 0.41 
and 0.27 respectively. These associated histograms and statistics are depicted in Figures 
24 - 29. 




Investment - 


JnJisJ 


Histogram Statistic Preferences | Options | 


Statistics I 

Result I 

Number of Trials 

1000 

Mean 

2.480384E+008 

Median 

2.480941 E+008 

Standard Deviation 

4.24G487E+005 

Variance 

1.803265E+011 

Coefficient of Variation 

0.0017 

Maximum 

2.489396E+008 

Minimum 

2.461307E+008 

Range 

2.808873E+006 

Skevbness 

■0.7525 

Kurtosis 

0.6814 

25*4 Percentile 

2.477840E-q)08 

75*4 Percentile 

2.483402E-q)08 

Percentage Error Precision at 95*4 Confidence 

0.0106*4 


Figure 23. Volatility of the returns on the $305 million Software Investment. 

The forecast statistics summarizes the distribution of the forecast 
values in terms of the four moments of a distribution. 



Type: Two-Tail. Lower: -Infinity. Upper: Infinity, Certainty: 100.0000% 


Figure 24. Probability of Requirements Increases during the Specification 

Phase. 
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The frequency histogram (Figure 24 on previous page) shows the frequency counts of 
values occurring in the total number of trials simulated. The vertical bars shows the 
frequency of a particular probability of requirements increase during the specification 
phase occurring out of the total number of trials, while the cumulative frequency depicted 
by the smooth line shows the total probability of all values at and below the maximum 
probability of requirements increase during the specification occurring in the forecast. 


fE Probability of Requirements liKMasesi 




Histogram Statistic | Preferences | Options | 


Statistics 1 

Result 1 

Number of Trials 

1000 

Mean 

0.1567 

Median 

0.1566 

Standard Deviation 

0.0014 

Variance 

0.0000 

Coefficient of Variation 

0.0089 

Maximum 

0.1603 

Minimum 

0.1538 

Range 

0.0065 

Skewness 

0.2620 

Kurtosis 

-0.6317 

25*4 Percentile 

0.1557 

75% Percentile 

0.1577 

Percentage Error Precision at 95*4 Confidence 

0.0553*4 


Figure 25. Statistics of Requirements Increases during Specification Phase. 

The forecast statistics summarizes the distrihution of the forecast 
values of the prohahility of requirements increases during the 
specification phase in terms of the four moments of a distrihution. 


76 








Type: Two-Tail, Lower -Infinity, Upper Infinity. Certainty: 100.0000% 


Figure 26. Probability of Requirements Increases during the Development 
Phase. The frequency histogram (Figure 26 on previous page) shows 
the frequency counts of values occurring in the total number of trials 
simulated. The vertical bars shows the frequency of a particular 
probability of requirements increase during the development phase 
occurring out of the total number of trials, while the cumulative 
frequency depicted by the smooth line shows the total probability of all 
values at and below the maximum probability of requirements 
increase during the development phase occurring in the forecast. 



Figure 27. Statistics of Requirements Increases during Development Phase. 

The forecast statistics summarizes the distribution of the forecast 
values of the probability of requirements increases during the 
development phase in terms of the four moments of a distribution. 
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Type Two-Tail, Lower; -Infinity. Upper Infinity, Certainty; 100.0000% 

Figure 28. Probability of Requirements Increases during the Validation 
Phase. The frequency histogram (Figure 28 on previous page) shows 
the frequency counts of values occurring in the total number of trials 
simulated. The vertical bars shows the frequency of a particular 
probability of requirements increase during the validation phase 
occurring out of the total number of trials, while the cumulative 
frequency depicted by the smooth line shows the total probability of all 
values at and below the maximum probability of requirements 
increase during the validation phase occurring in the forecast. 




Probability of Requirements Increases during V 


^nJ xJ 


Histogram Statistic | Preferences | Options | 



Statistics 1 

Result 1 

Number of Trials 

1000 

Mean 

0.2691 

Median 

0.2691 

Standard Deviation 

0.0109 

Variance 

0.0001 

Coefficient of Variation 

0.0407 

Maximum 

0.2953 

Minimum 

0.2432 

Range 

0.0521 

Skewness 

-0.0263 

Kurtosis 

-0.7022 

25% Percentile 

0.2612 

75% Percentile 

0.2777 

Percentage Error Precision at 95% Confidence 

0.2521% 


Figure 29. Statistics of Requirements Increases during Validation Phase. The 
forecast statistics summarizes the distribution of the forecast values of 
the probability of requirements increases during the validation phase 
in terms of the four moments of a distribution. 
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H. INTERPRETING THE FORECAST RESULTS 


To obtain the charts above, we examined two of the three variables in question - 
requirements increases during the specification and requirements increases during the 
development phase. We selected the Lognormal distribution from the Risk Simulator 
Tool due to the multiplicative impact of the variables in question on the overall volatility 
of the software investment. 

The four moments described as the first, second, third and fourth moments which 
the forecast statistics summarizes are respective measures of the following parameters 
[14]. 

1. Measure of Returns 

2. Measure of Risk 

3. Measure of Skewness 

4. Measure of Kurtosis 

All these four moments provide a more comprehensive view of the project under 
analysis and are explained in detail in [14] as follows: The first moment depicted by the 
mean value measures the expected rate of return on the investment effort, by measuring 
the location of the different scenarios of the investment effort and the average possibility 
of the outcomes. The second moment depicted by mathematical functions such as 
standard deviation, variance, coefficient of variation and volatility provides a risk 
measure of the investment effort through a measure of the variability of the variable 
(spread of the distribution). The third moment provides a measure of skewness by taking 
into account the differences in the skewness of a distribution should a situation arise that 
two or more different investment projects under consideration have the same mean and 
standard deviation. The fourth moment provides a measure of the peakedness (kurtosis) 
of the distribution as a way of differentiating between two or more investment projects 
with similar mean, risk and skewness distribution (first, second and third moments). This 
means that although the risk and returns are identical, the probability of extreme and 
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catastrophic events oeeurring is higher for a high kurtosis distribution and a higher exeess 
kurtosis value indieates that the downside risks are higher. 

The volatility of requirements ean be observed in Figures 24, 26 and 28. The 
kurtosis is given by -0.6317, -0.7401 and -0.7022 for the speeifieation, development and 
validation phases respeetively. From our observations, we ean see that that the probability 
of requirements ehanges are mueh higher during the development phase. 

1. APPLICATION OF DEMPSTER-SHAFER THEORY 

At this point sinee we have determined the volatility of our software projeet as 
well as the assoeiated probabilities of the impaet of the risk of requirements ehanges, the 
exeeutive would like to further reduee the uneertainty in the objeetive probability 
estimates to better estimate the impaet of requirements ehanges, so that the appropriate 
option ean be ereated where applieable. Therefore we now resort to the DST approaeh. 

We begin by partitioning our problem of requirements inereases based on the 
phases of the software engineering (SE) proeess, so as to identify the impaet of the 
requirements inereases on the different phases of the SE proeess on the value of the asset. 
We aeeomplish this by eolleeting additional evidenee using the Delphi Method to gather 
more information based on the expert opinions of two independent experts. 

We then develop our frame of diseemment whieh eonsists of the set of mutually 
exelusive alternatives or possibilities whieh the risks of requirements inereases pose to 
our eurrent software investment effort. Therefore 0 would be the set eonsisting of all the 
possible phases in which the risk of requirements inereases affeet the value of the 
software investment. We denote these as follows 

• Requirements inerease during Software Speeifieation (SS) 

• Requirements inerease during Software Development (SD) 

• Requirements inerease during Software Validation (SV) 

Thus our frame of diseemment is 0 = {SS, SD, SV} 
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Moving along the lines of our example, suppose independent expert’s #1 
determines subjeetively that the initial estimates obtained from the historieal sourees are 
0.90 reliable, i.e. requirements inereases during the development phase eontributed the 
most to the volatility of the software-investment. Therefore under DST belief funetion we 
ean establish a 0.9 degree of belief that inereases in requirements during the development 
phase would lead to a inerease in eosts, and a 0 (not 0.1) degree of belief that inerease in 
requirements during the development phase would not lead to inereases in costs. (Under 
classical probability theory, the software executive can say that the independent expert’s 
Delphi Method is 0.1 unreliable.) 

The 0.9 and the 0, (which do not add to 1) together constitute a “belief function”, 
where the “0” degree of belief does not imply that independent expert #1 is sure that 
increases in requirements during the development phase would not lead to cost increases, 
as is the case in classical probability, but simply implying that the historical estimates 
gives independent expert #1 no reason to belief that increases in requirements during the 
development phase would not lead to cost increases. 

We can then develop our propositions based on the following axioms 

{m)Bel (0) = O 

(B2) Bel (0) = 1 

(B3) Given a set of subsets of 0 , Ai.An 

Bel (Ai U.U An) > 2 Bel (A,) - E Bel (A, n Aj) + ...+ {-iT^Bel (A. n. (1 An) 

Therefore independent expert #l’s belief Bel{A) in any subset 0 could then be given as 
follows: 

5e/({SD}) = 0.9 
BeliiSD, SS, SV})= 1 

Note that the belief function basically conveys the same information as the basic 
probability assignments as it represents evidence in favor of the proposition. For example 
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we can also assign basic probability assignments based on Independent experts #1 
opinion as shown below since each element is the sum of the probabilities of the sets 
enclosed by the element, (the concept of masses), since for example SS could further 
consist of interface and operating systems requirements as its subset both of which have 
their associated probabilities. 

m({SD}) = 0.9 

m({SD, SS, SV}) = 0.1 

We can then compute plausibility using the equation below 
Pl{A) = \-Bel(K) 

To further compute Bel (A ) as a doubt function 
Dou({SD}) = 5e/({SS, SV})= 0 
Dou({SD, SS, SV}) = Bel (0) = 0 

Therefore our upper probability function in the form of plausibility for each subset is 

P/({SD})=1-Dou({SD})= 1 

P/({SD,SS,SV}) =1 - Dou({SS,SD,SV}) = 1 

To illustrate the concept of combination of information from an additional source 
of information (expert opinion), we present two scenarios. The first scenario depicts the 
second expert opinion providing information consistent with the objective estimates and 
the second providing conflicting evidence against the objective estimates. 

In our first scenario, say the independent expert #2’s also has a similar subjective 
probability of 0.90 on the reliability of requirements increases affecting the specification 
phase based on the assessment of historical estimates. Thus based on the second source of 
information (independent expert #2’s Delphi Method) we can reasonably assign 
probability assignments to each of the elements as follows: 

m({SD}) = 0.9 

m({SD, SS, SV}) = 0.1 
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and also develop the corresponding belief function 
5eZ({SD}) = 0.9 
BeliiSD, SS, SV})= 1 

Similarly the computation of plausibility mimics the plausibility computation above. 
Dou({SD}) = 5e/({SS, SV})= 0 
Dou({SD, SS, SV}) = Bel (0) = 0 

Therefore our upper probability function in the form of plausibility for each subset is 
/’/({SD})=1-Dou({SD})= 1 
P/({SS,SD,SV}) =1 - Dou({SS,SD,SV}) = 1 

We now attempt to combine the information from both independent expert #1 ’s & 
#2’s Delphi Method in order to obtain our evidence functions. We begin this process by 
creating a matrix and multiplying the masses from the associated column and row of both 
sources of information in order to calculate the combined basic probability assignment 
for a particular cell. In other words to compute the orthogonal sum in the form m = 
m, © where mi and m 2 represent the basic probability on our frame of discernment 
0. Table 5 below shows the basic template we use for our computations. 




IndpendentEipeitl 


CostlncieeseFeclois 

SD ml 

SD.SS,SV ml 

Independent EipefI 2 

|SD| in2 

ml[|SD|]'m2(|SD|| 

ml(|SD,SS,SV||'m2(|SD|) 

|SD.SS.SVi m2 

ml||SD|rm2[|SD.SS.SV|1 

ml[|SD.SS.SV|rm2[|SD.SS.SV|l 


Table 5. Matrix Template Used To Compute Orthogonal Sum Of Basic Probability 

Assignments 


We apply this template to our scenario and are able to come up with the combined 
basic probability assignments as shown in the matrix below (Table 6) by using Eqn 19 
above given by the equation on the next page. 
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Indpendent Eipert 1 


Cost Increase Factors 

SD ml: 0.30 

SD.SS.SV ml = 0.1 

Independent Eipeit 2 

|SD| m2:D.3D 

0.31 

0.03 

ISD.SS.SVI rnZ:D.I 

0.03 

0.01 


Table 6. Orthogonal Sum Of Basic Probability Assignments For Consistent Evidence 


X'^|(Ai)"*2(A2) 

A j A2 — A 

Since the information provided by independent expert #Vs & #2’s Delphi Method is 
consistent. We obtain the following evidence functions 

mi © m 2 ({SD}) = 0.81 + 0.09 + 0.09 = 0.99 

mi©m2({SD, SS,SV}) = 0.01 

Therefore given the evidence presented by mi and m 2 , we can say the most 
probable belief for this universe of discourse is consistent with the probability 
assignments of both belief functions and no further action is needed besides determining 
the option to address the risk presented. 

In the event that the sources of information contradict each other as is the case in 
our second scenario, such that while independent expert #Vs evidence indicates a 0.9 
probability of belief, independent expert #2’s evidence indicates a 0.7 probability of 
belief, we assign the following probability assignments. 

m({SD}) = 0.7 

m({SD, SS, SV}) = 0.3 

and also develop the corresponding belief function 
5e/({SD}) = 0.7 
BeliiSD, SS, SV})= 1 

Similarly the computation of plausibility mimics the plausibility computation above. 
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Dou({SD}) = 5e/({SS, SV})= 0 
Dou({SD, SS, SV}) = Bel (0) = 0 

Therefore our upper probability funetion in the form of plausibility for each subset is 
P/({SD})=1-Dou({SD})= 1 
/’/({SS,SD,SV}) =1 - Dou({SS,SD,SV}) = 1 

Thus we are able to come up with the following matrix of combined probabilities. 




IndpendentEipertI 


Cost Incieese Factors 

SD ml: 0.30 

SD.SS.Sy ml:0.1 

Independent Eipeit 2 

|SD| in2:D.7D 

0.03 

0.07 

ISD.SS.SVI in2:D.3D 

0.27 

0.03 


Table 7. Orthogonal Sum Of Basic Probability Assignments For Conflicting Evidence 


We obtain the resulting evidence functions; 


mi © m2({SD}) = 0.63 + 0.27 + 0.07 = 0.97 
mi©m2({SD, SS,SV}) = 0.03 

To examine the degree of conflict we revisit Eqn 17 above depicted as: 
mj(B)m2(C) 


mn{^) = X 


BnC=A 


\-K 


-when A ^ 0 


where 

K= J^m,(B)m2(C) 

BoC=0 


However since we have no null intersections, the sum of all such sets is in our matrix in 
Table 7 is equal to 0 meaning there is no conflict in the two expert’s opinion. 

The plausibility (certainty) of this belief is computed based on the following doubt 
functions; 
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Dou({SD, SS, SV}) = Bel (0) = 0 
Dou({SD}) = Bel (SS, SV) = 0 
Thus plausibility is as follows: 

P/({SD, SS, SV) = 1- Dou({SD, SS, SV}) = 1 
P/({SD) = l-Dou({SD }) = 1 

The resulting value of 1 in both oases denotes the upper probability funotion and 
expresses how muoh oonfidenoe there is that ({SD, SS, SV}) and ({SD}) risk oontributes 
to volatility. Based on the information derived from the matrix we oan establish the 
following joint beliefs 

Joint Belief of Independent expert #1 and #2 estimates in {SD} is 0.97 

Joint Belief of Independent expert #I and #2 estimates in {SD, SS, SV} is 1 

Within the oontext of our assumptions, the joint belief in {SD} of 97% infers that 
both independent experts have a high belief in the propositions surrounding requirements 
inereases during the development phase. Any variations between inferred probability 
assignments based on the mass of evidenee under this joint belief and our initial volatility 
estimates would re fleet ineonsisteneies. These variations are eaptured and used to refine 
the initial probability estimates to refieet the new “findings” whieh are then modeled 
using a Monte Carlo simulation to derive new estimates for the projeet and an overall 
volatility for the projeet. 

For example, in our seenario, say after we revise our probability estimates to 
reflect the findings based on the application of the approach and model these new 
estimates, we obtain a new volatility estimate of 0.0012 which infers a reduction in our 
original volatility estimate of 0.0017. This new volatility of the software project would 
result in an increase of NPV from the initial computed estimate of $248,170,302.58 to 
$248,762,936.51 meaning that since our project is less volatile than we initially assumed, 
our NPV would increase to reflect the more accurate valuation in light of the revised 
volatility of our investment effort. Figure 30 on the next page depicts the histogram 
reflecting our new data under this scenario. 
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Figure 30. Revised NPV Returns on $305 million Software Investment. The 
frequency histogram shows the frequency counts of values occurring 
in the total number of trials simulated. The vertical bars shows the 
frequency of the revised NPV returns occurring out of the total 
number of trials, while the cumulative frequency depicted by the 
smooth line shows the total probability of all values at and below the 
maximum revised NPV occurring in the forecast. 



Figure 31. New NPV Returns on $305 million Software Investment based on 
refined probabilities. The forecast statistics summarizes the 
distribution of the revised forecast values in terms of the four moments 

of a distribution. 
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J. ANALYSIS 

We can then repeat this whole process as often as needed until we feel confident 
that we have reduced all our uncertainties as much as possible. It is also important to note 
that other special cases might also exist related to categorical, completely non- 
probabilistic information. In these cases, while one is 100% confident in the reliability of 
the source of information or evidence, the information is still lacking in completely 
answering our question. For example in hypothetical example drawn along the lines of 
reasoning in [51] there might be conclusive evidence, that a change in a software 
requirement might affect one of three places in the overall software, without any clue as 
to which one. This calls for a belief function that assigns a degree of belief of 100% to 
the three places as a set, but a degree of belief of 0% to each of the three individually. 

In our example, we are dealt with a question that has only two answers (whether 
or not increases in requirements are associated with increases in costs). In the case that a 
question has more than two answers, belief functions can be derived by determining the 
degree of belief for each answer and for each set of answers, although the belief function 
may be very complex if the number of answers (or the size of the “frame”) is large. Now 
that we have successfully depicted how we estimate volatility, we proceed to the next 
phase of our process, which is the development of the “real option” to mitigate the risks. 
We depict this in the following chapter. 
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V. REAL OPTIONS FRAMEWORK FOR SOFTWARE 

INVESTMENTS 


.. .there is nothing more difficult to take in hand, more perilous to 
conduct, or more uncertain in its success than to take the lead in the 
introduction of a new order of thing... 

(Niccolo Machiavelli, philosopher, 1469-1527) 


A. INTRODUCTION 

The underlying premise of the Real Options (RO) approach is the ability to li nk 
uncertainty to the value of flexibility within the real options framework. In the previous 
two chapters, we addressed the critical issue of uncertainty identification and 
quantification in order to determine volatility, a key input into the real options 
framework. We also established that uncertainty affects decision associated with 
software-related capital investments. In this chapter, we build on the body of knowledge 
established thus far and proceed to establish an RO framework that could be used to 
address software-related capital investments under the underlying assumption that when 
properly utilized by a well empowered management team (another precondition for the 
application of RO theory), it would better guide the decision-making process. However, 
before we establish the framework we first discuss the issue of valuing the underlying 
software investment. 

B. BASE CASE VALUATION OF THE UNDERLYING ASSET 

In Real Options risk-based analysis, the second most important measure besides 
volatility estimation is the valuation of the underlying asset being considered for 
acquisition. This is typically the first activity accomplished in a software-related capital 
investment effort. The RO approach calls for the existence of a financial model before it 
could be applied. This involves presenting or developing a financial model of the 
underlying asset, as the real options approach requires the use of an existing discounted 
cash flow model despite their limitations. In a discussion of the limitations of some of the 
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investment valuation approaehes in the assessment of previous work done in Chapter 2, it 
was asserted that the limitations of the existing valuation models were primarily due to 
the laek of eonsideration of managerial flexibility. Net Present Value (NPV) is measured 
in today’s dollars and its eomputation is based on the prineiple of discounting in whieh 
all projeeted future eash flows representing estimated eosts, eost savings, and revenues at 
various points during the useful lifetime of the projeet, are discounted utilizing a diseount 
rate. This diseount rate eaptures the opportunity cost and the eost of money (based on 
interest rates) assoeiated with the underlying investment, back to the present time under 
the assumption that one dollar today is worth (1 + r)' dollars at time t in the future [52]. 

The traditional NPV formula is given as 



-Co 


Similarly, a diseounted eash flow (DCF) analysis ean be eomputed as 


DCF = 


CF, 

(l + r)‘ 


+ 


CF, 

(1 + r)^ 


+ 


,+ 


CF^ 

(1 + r) 


where CF = Cash Flow, r = diseount rate and t = time 

It must however be emphasized that in performing a diseounted eash flow 
analysis the diseounting faetor is coneerned with two things: the diseounting eonvention 
and the diseount rate itself. There are several diseounting eonventions ranging from 
eontinuous versus diserete, midyear versus end-of-year, to beginning versus end-of- 
period. It is eritieal to understand the eonvention being used in order to maintain 
eonsisteney in results as eaeh of the diseounting eonventions produee different results. 
The seeond faetor, diseount rate is normally the weighted average eost of eapital 
(WACC) for the eash flow series. However, this is eonsidered to be somewhat 
problematie beeause of risks ranging from organizational eapital strueture and investment 
policies to the flaws in the Capital Asset Pricing Model (CAPM). 

As detailed financial data is not always readily or publicly available for U.S. 
government investment project, we rely on guidance provided by the U.S. Office of 
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Management and Budget (0MB) for NPV eomputation [53]. 0MB states that the 
standard eriterion for deeiding whether a government program ean be justified on 
eeonomie prineiples is based on the diseounted monetized value of expected net benefits 
(i.e., benefits minus costs). The guidance further states that “social” net benefits and not 
the benefits and costs to the federal government, should be the basis for evaluating 
government programs or policies that have effects on private citizens or other levels of 
government. Consequently, the emphasis on “gained” social benefits either real or 
perceived needs to be factored into NPV computations in software related capital 
investments funded and operated by the government. In the case of non-measurable 
benefits and costs, further analysis is needed to guide decision makers when they have to 
weigh the net non-measurable benefits against net measurable costs. 

From a DoD perspective, the perceived value created of any software investment 
is gauged from the ability of the investment to contribute meaningfully to the DoD’s 
Strategic, Operational and Tactical objectives, be it for the war-fighter or the running of 
the various agencies supporting the war-fighter. While it might be somewhat difficult to 
assign a monetary value to this benefit, attempts could be made to either estimate or 
employ qualitative forecasting techniques in the absence of any current or historical data. 
Expert opinions or relative estimates based on comparable proxies could also be 
explored. Specifically, the value of DoD software or technological investment could be 
gauged and estimated on how the investments contributes to or enhances mission or 
operational readiness, value received in the form of reduction in casualties/injuries 
amongst U.S. service members, estimated net annual recurring savings, etc. In the case of 
weapons systems development, the expected benefits realized prior to the development of 
counter-measures by potential adversaries also need to be factored into the NPV 
computation. Regardless of the factors explored, it must be emphasized that this is a non¬ 
trivial task. 

For the purposes of our study, we have chosen to employ relative NPV valuation 
as depicted in our example in the previous chapter. Under traditional NPV analysis, a 
project with a positive NPV increases the wealth of the firm, that is, the total value 
generated through the project’s lifetime is superior to the cost of financing it and a higher 
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NPV is always preferable to a lower NPV, while a negative NPV represents an 
unaeeeptable investment. This traditional NPV formula ean be tailored to speoifieally suit 
a software development projeet as outlined in [52] in whieh the NPV for a software 
development projeet is eomputed in terms of five high-level determinants and the 
equation outlined as follows: 


NPV= Z 


{\ + ry 


where 


z 


v(l + 0'. 


>C 


and 


M= y 


M. 


{\+ry 


where the standard assumption in [52] is (C -M) is always positive and 

I is the (initial) development cost. It represents the total present value of all 
negative eash flows from the time the deeision to invest is made to the time of the first 
major positive eash flow (time T) with development eost and development time are 
positively eorrelated 

T is the (initial) development time or time to market. It is defined as the elapsed 
time between the eommitment to invest in the projeet and the time of its first major 
positive eash flow. This period eovers aetivities leading to the deployment of the end 
produet. 

C is the asset value. It represents the total value of the positive eash flows that the 
projeet is expeeted to generate during its lifetime, ealeulated at time T. Asset value 
mainly eonsists of the revenues from sales in the ease of Foreign Military Sales, direet 
eost savings from using the end produet and soeial benefits derived from aequisition of 
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the software produet. Direct product costs, whieh represent the expenses ineurred in 
proportion to the revenues generated, are dedueted from asset value. 

M is the operation cost. It is the total value of all negative eash flows of the 
operation phase, ealeulated at time T. This amount eonsists mainly of regular 
maintenanee eosts and pre-planned future investment outlays. Note that direet produet 
eosts (whieh are dedueted from asset value) are not ineluded here. 

r is the rate at whieh all future eash flows are to be diseounted (the discount rate). 


Teehnieally speaking we ean reasonably eonelude that NPV is a Real Options 
approaeh that assumes no flexibility in the deeision making proeess. When we faetor in 
flexibility in the form of Real Options ereated to hedge risk is faetored into NPV 
eomputation, we arrive at the following formula 

NPV=Z -l + Q. 

{\+ry 

where 

Q is the flexibility (option) premium. It measures the eontribution of the projeet’s 
inherent strategie flexibility to its base NPV under uneertain eonditions. For example, the 
ability to delay the eommitment of eertain resourees, or to ehange the maintenanee 
sehedule. 

The determination of the flexibility premium is dependent on several faetors. 
Before we outline its determination, we need to revisit the issue of RO and how they 
relate to software related eapital investments. First, we outline the approaeh to refining 
the value of our underlying asset. 

C. REFINING ASSET VALUE USING MONTE CARLO SIMULATION 

Traditional valuation methods are usually a statie single point estimate whieh is 
insuffieient for eredible analysis. Consequently to better or aeeurately estimate the aetual 
value of a partieular projeet, a Monte Carlo simulation would need to be run [14]. 
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However, before the Monte Carlo simulation ean be run, the key input variables whieh 
are deemed highly uneertain in the future need to be identified. This could be 
accomplished by conducting a sensitivity analysis on the valuation model. Once the key 
input variables have been identified, the following steps as outlined in [14] are carried out 
during the Monte Carlo simulation. 

1) A probability distribution is defined for each of the key inputs which 
underlie the cash flows. Initial estimates of the key input parameters, are 
obtained using either historical data or subject matter experts or a 
combination of both. 

2) An outcome is drawn in each simulation from each distribution and the 
present value of the cash flows estimated based on the draws 

3) The expected value of the project is determined from the mean of the 
distribution of the expected cash flows which is computed after several 
simulations and the standard deviation or the variance of the distribution 
of the cash flows can be used to value the project. 

D. REAL OPTIONS 

An option gives its owner or option holder the right, but not the obligation, to buy 
or sell an asset at a certain pre-negotiated price and the most obvious determinant of an 
option’s value is its intrinsic value, or what it would be worth if it were immediately 
exercised [6]. This amount, defined as the option price less the exercise price, ultimately 
determines how much money the option holder makes. An option can still be valuable 
even if it has no intrinsic value because the possibility exists that the option could be 
profitably exercised in the future. The value of this possibility is an option’s time value 
and is determined by three factors: volatility, length of time before an option expires, and 
risk-free-rate [6]. In other words, an option is essentially a precise contract between the 
software executive and the contractor (in the case of a government acquisition) to buy or 
sell a specific capability known as the options underlying instrument. In the case of 
software investment options from both a managerial and technical perspective, the 
underlying instrument is the software itself The “contract” has an associated strike price 
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at which the contract or option may be “exercised” or acted on as well as an expiration 
date. In general, options can be grouped into two categories: “calls” and “puts”, both of 
which could be bought and sold. 

1. Buying Call and Put Options 

In the event that a software executive buys a call option, the software executive 
would have the right but not the obligation to buy the underlying instrument at the strike 
price on or before the expiration date which would more than likely should be tied to the 
software development or delivery schedule in the case of software acquisitions. In the 
event that a put option is bought, the software executive would have the right but not the 
obligation to sell the underlying instrument on or before expiration of the option. 

2. Selling Call and Put Options 

On the other hand, if the software executive decides to sell a call option, the 
executive becomes obligated to sell the option or underlying interest at the strike price to 
the buyer should the buyer so decide and in the case that the software executive decides 
to sell a put option, the software executive becomes obligated to buy the underlying 
interest. In this case, since the software executive is the “writer” of the options, the 
executive would really has no control over whether or not the option is exercised. 

Both cases involve a fee called the “premium”, with the purchase price of the 
option being the premium in the event that an option is bought, and the selling price 
being the amount that is received. Once the appropriate options have been identified, they 
need to be valued accordingly. 

E. IDENTIFYING STRATEGIC REAL OPTIONS 

As discussed in the previous chapter, we implement our methodology which 
incorporates the Dempster-Shafer theory (DST) approach to volatility estimation which 
generally involves solving two related problems. 

1) First, we must sort the uncertainties in the problem into a priori 

independent items of evidence. 
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2) Second, we must carry out the Dempster's rule computationally by 
combining the evidence obtained from all sources 

We then model our results stochastically using the Risk Simulator and determine 
the correlation of variability. Once we have determined the volatility of our software 
investment effort, we can now identify possible options which we could used to manage 
the risk associated with the software investment effort. Generally speaking, over the 
course of the examination of a software investment effort, several independent options or 
combinations of options (compound options) can be identified to mitigate or hedge risk 
as it emerges. While options come in various forms such as American, European, Asian, 
and Bermudian option, the underlying concepts between each of these forms remain the 
same with the only differences being in the period or time of execution of the options. To 
applying options theory in this study, we explore various types of options for analyzing 
and addressing uncertainties associated with software-related capital investments which 
we broadly categorize into 

1) Expand/growth options, 

2) Wait/Deferment options and 

3) Contract/Switch/Abandon options. 

We depict these categories in Eigure 32 below. 
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Real 

Option 

Category 


Real 

Option 

Type 


Description and Example 


Scale 

up 


Option to scale up through cost effective sequential investments as 
knowledge of the product increases 


Expand / 
Growth 


Switch 

up 


A flexibility option to switch products, processes given a shift in underlying 
price of input and output demands 


Scope 

up 


Investment in proprietary assets of one industry enables company to enter! 
another industry cost effectively . Link and Leverage I 


Wait/ 

Defer 

-- 

Study/ 

start 

Delay Investment until more information or skill is acquired.! 
e.g. Introduction of new requirements 1 

h - 


1 



Scale 

down 


Shrink or shut down a project part way through if new information | 
changes the expected payoffs, e.g. Introduction of new requirements 


Contract / 
Switch / 
Abandon 


Switch 

down 


Scope 

down 




V, 


Switch to more cost effective and flexible assets as new information 
is obtained, e.g. switch from custom development to COTS 


Limit scope of (or abandon) software project when there is no further 
potential in the business opportunity is meant to address 


Figure 32. Sample Options to Address Software Investments (From: [6]). 


In addition we also explore variants or combinations of options in the form of 
Sequential and Simultaneous compound options and explore a valuation methodology for 
the options. 

F. PARTITIONING: DECOMPOSING THE SOFTWARE SOLUTION 


To take advantage of the options identified above, we address the issue of 
software design. From a Real Options perspective, the decomposition of the software into 
components, modules or subsystems serves to introduce flexibility which the software 
executive or program manager could exploit and benefit from. Software design is a key 
activity aimed at conceiving how a software solution would solve a particular problem 
and it involves dealing with critical design decisions such as the decomposition/ 
partitioning of the software product under consideration into components to make its 
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acquisition and development more manageable. Given the impaet whieh design deeisions 
have on a software produet, managerial eoneerns/uneertainties ranging from work-foree 
eapability to maturation/market availability of the proposed teehnology and teehnieal 
eoneerns/uneertainties ranging from maintainability to reusability should be faetored into 
design deeisions. It must be noted that while we break this down into both managerial 
and teehnieal eoneems, these eoneerns ean and do overlap. In the ease of large-seale 
eapital-intensive software-investment efforts, deeomposing or partitioning the overall 
software produet into well-defined, independent eomponents based on funetionality has 
the benefit of allowing eaeh of the independent eomponents to be developed by different 
vendors with the understanding that the ultimate end-goal would be to integrate eaeh of 
the independent eomponents to arrive at a fully funetional software produet that meets the 
eustomers needs. However it must be noted that design deeisions are mostly made under 
uneertainty, and while these uneertainties might have been faetored in the design during 
the high-level funetional deeomposition of the software, deeomposition does not imply 
that uneertainty no longer exists, but merely redueed. This is beeause by virtue of 
software being an intangible artifaet, it is impraetieable to prediet or have aeeess to 
perfeet information to key faetors assoeiated with the operation of the software when 
deployed at the time of the design. Henee uneertainties might still remain in a eomponent 
or aeross eomponents after deeomposition stemming from both development and 
operational perspeetive eoneerns. Furthermore, while it is impossible to totally eliminate 
uneertainty, reduetion strategies eould however be employed sueh as the use of high 
fidelity modeling and simulation tools explieitly tailored to address uneertainty. By 
modeling and simulating the areas of eoneem within and aeross the eomponents 
uneertainty, espeeially uneertainty arising from requirements uneertainty, operational 
uneertainty and uneertainty about the emergent attributes eould be either redueed or 
resolved and ultimately help inerease the degree of eertainty assoeiated with the 
deeomposed eomponents and the overall design of the software solution. 

Given the unresolved issue of requirements uneertainty in our example in the 
previous ehapter, the software exeeutive needs to figure out the development approaeh to 
utilize for the software development effort. After eonsidering several software 
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development options, the exeeutive deeides a stage-gate development approaeh based on 
eoneepts of both iterative development approaehes and standard portfolio modeling 
teehniques would be the most optimal development approaeh. 

The approaeh would allow the software exeeutive to partition/deeompose the 
proposed software into different independent subsystems or eomponents (e.g., the flight 
eontrol subsystem and the avionies subsystem) and form a “portfolio” of the subsystems 
around the solution spaee (the software that needs to be developed). By doing this, the 
development of the subsystems ean begin at the same time using iterative development 
teehniques to guide eaeh subsystem towards sueeess while at the same time deferring 
subsystems that still faee requirements uneertainty. The iterative eoneepts employed 
would faeilitate shorter feedbaek eyeles by eyeling through the development phases, from 
gathering requirements to delivering funetionality in a working release. 

In essenee this approaeh allows the software exeeutive to reeast the KC-X 
software program as a series of options to start and defer the development of a subsystem 
when requirements uneertainty is eneountered, realloeate resourees and then resume as 
uneertainty beeomes resolved a strategy whieh supports the following two propositions 
[54]. 

• Some projeets that look attraetive on a full investment basis may beeome even 
more attraetive if the projeet is partitioned/deeomposed into eomponents beeause 
we are able to reduee downside risk at the lowest possible level. 


While we made the assumption that our software investment was attraetive in our 
example, if we had made the assumption that it was unattraetive, the seeond proposition 
below would also hold if we were to adopt the stage-gate-development approaeh 


• Some projeets that are unattraetive on a full investment basis may be value 
ereating if the firm ean invest in stages. 

Therefore by embraeing a stage-gate development approaeh, the exeeutive hopes 
to exploit the basie eoneepts of standard portfolio theory in order to reduee risk by 
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beginning the development of all the subsystems simultaneously with the hope that most 
of them would not be affeeted by requirements uneertainty and that in ease some of them 
are affeeted by requirements uneertainly, the eompletion of those subsystem ean be 
deferred until the uneertainty beeomes resolved, an approaeh whieh is basieally an 
“Option” on an “Option”, with the options being the Option to Defer and the Option to 
Switeh. Thus we frame our Real Options as follows: 

1) Deferment Options: Option to defer the eompletion of a subsystem until 
more information beeomes available 

2) Switehing Options: Option to switeh resources to the development of other 
subsystems if the development of a subsystem they are working on has to 
be deferred until more information becomes available. 

Given the relationship between our options in this case, we can call them a 
Compound Option where the value of each option is dependent upon whether the 
previous option was exercised or not and could also provide a method of stress testing or 
sensitivity testing of the final results by treating the options as alternatives within the 
option pricing model [14], Moving along the lines of our example in the previous chapter, 
we can then develop a strategy tree showcasing the options. We assume in our example 
that the software executive partitioned the solution space of the proposed software for the 
KC-X program into three independent systems capable of functioning independent of the 
other systems: the Flight Control system, the Avionics system and the Navigational 
system, which we denote as component A, B and C respectively, and assume that the 
development of all of these component start simultaneously. In the event that the 
development of any component needs to be stopped due to requirements uncertainty, our 
proposed uncertainty elicitation model in Chapter III (Figure 9) could be applied to 
attempt to resolve the uncertainty. This approach is useful in that it might also help 
identify requirements that had not been considered but have been uncovered due to the 
partitioning of the proposed software. Consequently we can develop a strategy tree based 
on seven possible strategies. The strategy tree is given in Figure 33 on the next page. 
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start development of 
components 
A, B and C 
simultaneously 


strategy A 


Uncertainty In 
component A 


strategy B 


Uncertainty In 
component B 


strategy C 


Uncertainty In 
component C 


strategy D 


■ Uncertainty In 

component A and B 


strategy E 


Uncertainty In 
component A and C 


strategy F 


Uncertainty In 
component B and C 




strategy G 



Uncertainty 
in A B and C 



Phase I 


Continue development 
of B and C and defer 
Component A 


Exit 

Do Nothing 


Phase 


Allocate resources from 
component A to develop 
component B and C 


Phase 


Switch back to development of 
component A when uncertainty is 
resolved and reallocate resources 
back from component B and C 
to A 


Exit 


Exit 


Stop after phase II 


Phase I 


Continue development 
of A and C and defer 
Component B 


Phase 


Allocate resources from 
component B to develop 
component A and C 


Exit 


Exit 


Phase III 


Switch back to development of 
component B when uncertainty is 
resolved and reallocate resources 
back from component A and C 
to B 


Exit 


Stop after phase III 


Phase I 


Continue development 
of A and B and defer 
Component C 


Phase ii 


Allocate resources from 
component A and B to develop 
component C 


Phase III 


Switch back to development of 
component A and B or A or B when 
uncertainty is resolved and 
reallocate resources back from 
C accordingly 


Exit 


Exit I 

Stop after phase III 


Exit 


Phase I 


Phase 


Continue development 
of C and defer 
Component A and B 


Allocate resources from 
component C to develop 
component A and B 


Exit 


Phase III 


Switch back to development of 
component C when uncertainty is 
resolved and reallocate resources 
back from component A and B 
to C 


Exit 


Stop after phase II 


Exit 


Phase I 


Phase i 


Allocate resources from 
component A and C to develop 
component B 


Phase I 


Switch back to development of 
component A and C or A or C when 
uncertainty is resolved and 
reallocate resources back from 
B accordingly 


Continue development 
of B and defer 
Component A and C 


Exit 


Exit 




Phase 1 



Continue development 
of A and defer 

Component B and C 




Phase 


Allocate resources from 
component B and C to develop 
component A 


Exit 


Stop after phase III 


Phase III 


Switch back to development of 
component B and C or B or C when 
uncertainty is resolved and 
reallocate resources back from 
A accordingly 


Exit 


Exit 


Exit 



Stop after phase III 



Phase 


Repartition Software Solution Space 


Exit 


Figure 33. Strategy Tree depicting strategic pathways for the Software 

Executive. Real Options framework around the KC-X software 
program and shows the different strategies the software 
executive or the decision maker can adopt to hedge risk in order 
to mitigate cost and schedule overruns. 
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G. ANALYSIS OF STRATEGIC OPTIONS 

Strategy A calls for the development of components B and C (Avionics system 
and the Navigational system) and deferring the development of component A (Flight 
Control system) using a “deferment option” until the uncertainty is resolved in 
component A. In this strategy, the resources that were initially allocated to component A 
are reallocated to components B and C using a “switching option” during the period of 
uncertainty resolution and are then allocated back to component A using a “switching 
option” once the uncertainty in A is resolved . 

Strategy B calls for the development of components A and C (Flight Control 
system and the Navigational system) and deferring the development of component B 
(Avionics system) using a “deferment option” until the uncertainty is resolved in 
component B. In this strategy, the resources that were initially allocated to component B 
are reallocated to components A and C using a “switching option” during the period of 
uncertainty resolution and are then allocated back to component B using a “switching 
option” once the uncertainty in B is resolved . 

Strategy C calls for the development of components A and B (Flight Control 
system and the Avionics system) and deferring the development of component C 
(Navigational system) using a “deferment option” until the uncertainty is resolved in 
component C. In this strategy, the resources that were initially allocated to component C 
are reallocated to components A and B using a “switching option” during the period of 
uncertainty resolution and are then allocated back to component C using a “switching 
option” once the uncertainty in C is resolved . 

Strategy D calls for the development of component C (Navigational system) and 
deferring the development of components A and B (Flight Control system and the 
Avionics system) using a “deferment option” until the uncertainty is resolved in 
components A and B. In this strategy, the resources that were initially allocated to 
components A and B are reallocated to component C using a “switching option” during 
the period of uncertainty resolution and are then allocated back to components A and/or 
B using a “switching option” once the uncertainty in A, B, or A and B is resolved . 
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Strategy E calls for the development of component B (Avionics system) and 
deferring the development of components A and C (Flight Control system and 
Navigational system) using a “deferment option” until the uncertainty is resolved in 
components A and C. In this strategy, the resources that were initially allocated to 
components A and C are reallocated to component B using a “switching option” during 
the period of uncertainty resolution and are then allocated back to components A and/or 
C using a “switching option” once the uncertainty in A, C, or A and C is resolved . 

Strategy F calls for the development of component A (Flight Control system) and 
deferring the development of components B and C (Avionics system and Navigational 
system) using a “deferment option” until the uncertainty is resolved in components B and 
C. In this strategy, the resources that were initially allocated to component B and C are 
reallocated to component A using a “switching option” during the period of uncertainty 
resolution and are then allocated back to components B and/or C using a “switching 
option” once the uncertainty in B, C, or B and C is resolved . 

Strategy G calls for repartitioning the software solution space along alternative 
partitions to the current set of partitions (Flight Control system. Avionics system and 
Navigational system) to see if there is sufficient information in any single subset to 
allocate resources and proceed with development of that subset while exercising a 
“deferment option” to wait until more information becomes available about the other 
subsets. 

Now that we have identified the appropriate options to hedge risk, the question 
that arises is how much does the software executive pay for the option? The following 
section addresses these concerns. 

H. MECHANICS OF OPTIONS VALUATION: OPTIONS PREMIUM 

Valuation plays a central part in any acquisition analysis. Options are usually 

valued based on the likelihood of the execution of the options. In general, there are three 

approaches to valuation, all of which have different outcomes depending upon which 

approach is used. These valuation approaches are discounted cash flow valuation/relative 

valuation, which are estimates based on the value of an asset by looking at the pricing of 
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'comparable' assets relative to a eommoii variable sueh as earnings, eash flows, book 
value or sales, and lastly eontingent elaim valuation. 

Sinee an option represents the right but not the obligation to make an investment, 
the payoff seheme to the option-holder is asymmetrie and the options are only exereised 
if they have a positive value and are left unexereised if worthless [6]. While there are 
several methods for eomputing and valuing real options sueh as employing the use of 
elosed-form models, partial differential equations, and the binomial approaeh. The 
binomial approaeh utilizes risk neutral probabilities elieits a great appeal due to its 
simplieity, ease of use and the ability to solve all forms of options (e.g., European, 
Ameriean). 

When a software exeeutive buys a eall option, the exeeutive expeets that the 
benefits derived from the option would translate into eost savings when the option is 
exereised in the future beeause at the time at whieh the option is bought, the software 
exeeutive is fully aware that the money spent on purehasing the option might not be 
reeovered if the option is not exereised or transferred to another similar government 
investment effort sinee teehnieally speaking the option eannot be sold for profit as a last 
resort, sinee the government is not in the business of making profit, but rather saving tax 
payers money. Therefore if the option is not exereised and is not transferred to another 
government entity that eould benefit from it, the eost of the premium must be faetored in 
the eomputation of realized benefits by subtraeting the premium (loss) from the projeeted 
realized benefits. On the other hand, the seller of the option has a net eredit [55] beeause 
of the premium that was reeeived for the option, sinee they would always keep the 
premium even if the option is never exereised. From a sellers perspeetive, who more 
often than not might be a private or eommereial organization, this eall option translates 
into a eost inerease thereby impaeting their profits if the option is exereised. Henee, they 
reeeive the option premium as a eompensation for their loss. 

Several faetors affeet the value of an option. These faetors eould range from the 
maturity of the option to the volatility of the underlying asset. Sinee premiums are not 
fixed and eonstantly ehange, the fluetuations refieet the eompromise between what the 

buyers are willing to pay for the option and what the seller is willing to reeeive for the 
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option with the buyers and sellers being the software executive and contractor 
respectively or vice-versa as the situation might warrant at which point an agreement is 
reached on the price of the transaction. 

Regardless, the options premium has two main components: intrinsic value and 
time value, both of which contribute to the valuation of the underlying software 
investment. In the case of call options, the call option is said to be “in-the-money” if the 
strike price of the call option is less than the current market price, meaning it becomes 
optimal for the option holder to exercise the option and benefit from factors such as 
realized cost savings in the form of inflation adjusted labor rates between the time the 
option was bought and the time the option is exercised. Likewise, in the case of a put 
option, if the strike price of the put option is greater than the current market price of the 
underlying interest, the put option is also said to be “in-the-money” [55]. If the events 
above do not hold, i.e. strike price is less than current market price for the call option or 
strike price is greater than current market price for the put option, the option is said to be 
“out-of-the-money” and should the strike price equal the current market rates, the option 
is say to be “at-the-money”. The amount by which a call or put option is in-the-money at 
any given moment is called its intrinsic value and any amount by which an options total 
premium exceeds the intrinsic value is the time value portion of the premium [55]. It is 
the time value portion of the options premium which is affected by factors such as the 
volatility of the software investment. 


'\**t«a* of 
opcioti 



Figure 34. Relationship between Stock Price and Call Option (From[6]). 

Thus in the case that an option is out-of-the-money, the option has no intrinsic 

value and the time value is the total option premium. As can be seen from the figure 
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above depicting the relationship between a call option and the stock (option) price, a call 
option’s intrinsic value increases as the stock price increases, but never falls below zero. 
This is analogous to a software investment. Since a call option’s time value increases as 
stock price volatility increases, so would a call option’s time value increase as the 
volatility associated with software investments increase because the longer an option 
holder has before expiration, the higher the probability that the value of the overall 
software investment will end up above the exercise price, thereby making options with 
longer “lives” more valuable than similar options with short lives. 

Furthermore the more volatile a stock, the higher the chance that the option will 
be profitable as can be seen from Figure 35 below depicting two scenarios, with one 
showing an option on a low-volatility stock with a narrow price distribution clustering 
around the exercise price and the second showing a similar option on a high-volatility 
stock with a wide price distribution. All things equal, then, higher stock price volatility 
translates into a higher time value [6]. 







Figure 35. 


Scenario A: Low stock price volatility Scenario B: High stock price 
volatility (From: [6]). 


Given the way an option is valued, we summarize the key points in Table 8 on the 
next page. 
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Call Option 

Put Option 

In-the-money 

Strike price of the call option 
is less than the current market 
price 

Strike price of the put option is 
greater than the current market 
price 

At-the-money 

Strike price of the call option 
is the same as the current 
market price 

Strike price of the put option is 
the same as the current market 
price 

Out-of-the-money 

Strike price of the call option 
is greater than the current 
market price 

Strike price of the put option is 
lower than the current market 
price 


Table 8. Value of an Option. 


I. VALUATION COMPUTATIONAL METHODS 

As mentioned earlier, to determine the value of our option created to hedge the 
risk presented by the uncertainties surrounding the software investment process requires 
the computation of several factors. The basic inputs are the present value of the 
underlying asset (5), present value of the implementation cost of the option (A), volatility 
(cv), time to expiration in years (7), risk free rate or rate of return in a riskless asset (r) 
and are summarized in Table 9 on the next page. 
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Symbol 

Real Option on Software 
Acquisitions Project 

Description 

S 

Value of underlying Asset (Asset 
Price) 

Current Value of expected cash flows. 

(Expected benefits realized from investing in 
the software effort (NPV)) 

K 

Exercise Price / Strike Price 

Price at which the created option would be 
realized (Investment Cost of investing in 
options, which is an estimation of the likely 
costs of accommodating changes) 

T 

Time-to-expiration 

The useful life of the option. (Time until the 
opportunity disappears or the maturity date of 
the option contract) 

r 

Risk-free Interest Rate 

Risk free interest rate relative to budget and 
schedule (Interest rate on EtS Treasury bonds) 

cv 

Volatility 

Etncertainty of the project value and 
fluctuations in the value of the requirements 
over a specified period of time (Volatility in 
requirements, cost estimation and schedule 
estimation based on DST of Evidence) 


Table 9. Factors Affecting Value of an Option. 

Although the Black-Scholes Formula is a more popular option pricing model, its 
underlying principle is based on European Call options, a key limitation since it cannot 
be used to price American Call options which allow the option holder to exercise the 
option any time up to the maturation of the option, hence the need for an alternative 
pricing method in the form of the Binomial Lattice. 

The binomial lattice approach could be based either on market replicating 
portfolios or risk neutral probabilities however the latter is much more easier to perform 
[14], since the risk neutral probability approach is simply a type of discrete simulation of 
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the cone of uncertainty and uses a "discrete-time" model of the varying price over time of 
the underlying financial instrument. 



Sd 

t = O t = 1 t = 2 


Figure 36. Binomial Lattice. 

So is the current asset value; the value moves up to Su with probability p and down to Sd 

with probability l-p in any time period. 

Option valuation is then computed via application of the risk neutrality assumption over 
the life of the option, as the price of the underlying instrument evolves [56]. A binomial 
lattice could be graphically represented in its simplest form as shown in the Figure 36 
above. It requires two basic computation; the up and down factor denoted by Su and Sd 
and the risk neutral probability measure with the up factor being the exponential function 
of the cash flow returns volatility multiplied by the square root of time steps {dt) (the time 
scale between steps) [14]. The risk neutral probability is simply computed as the ratio of 
the exponential function of the difference between the risk-free-rate and dividend 
multiplied by the time steps, less the down factor, to the difference between the up and 
down factors [14]. 

The binomial lattice approach to option valuation can be described as a three step 
process [56]. 

1) Asset Value tree generation 

2) Calculation of option value at each final node 
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3) Progressive ealeulation of option value at eaeh earlier node with the value 
at the first node is being the value of the option. 

A pre-requisite for its applieation is the existenee of at least two lattiees with one 
lattiee representing the underlying asset and the other representing the options valuation. 
It eould be used to visually deseribe priee movements over time, where the asset value 
ean move to one of two possible priees with assoeiated probabilities and the preeision of 
the results is eorrelated to the number of steps that run in the simulation. Given the 
faetors that are involved in computing the value of an option using the binomial lattice 
approach, the following formulas can be deduced: 


Su = e' 




( 20 ) 


- Sd 

p =- 

Su-Sd 

Sd = e-^^. 


( 21 ) 

..( 22 ) 


Therefore moving along the line of our example from the previous chapter, we can 
compute these factors based on the volatility of our software project of 0.0012, with a 
step size of 0.2 years (one year expiration divided by five steps ): 


Su = ^ 1.000536800340363 

Sd = e ^ = 0.999463487659644 

^0.03(0.2) _ 0.999463487659644 

^ 1.000536800340363 - 0.999463487659644 


= 6.1068396117618535 


Subject to the following underlying assumptions: 

• The present value of the underlying asset is $305 million 

• Implementation Cost is $55 million 


no 






• Net Present Value is $250 million 


• The risk free rate is the 3.0 

• Volatility of our project cv = is 0.0012 

• Duration is 12 Months (1 year) 

(We use coefficients of variation as a proxy of standard deviation because it is a relative 
measure of the standard deviation expressed as a percentage of the mean.) 

For the purposes of our example, we run the Super Lattice Solver 3.0 provided by 
Real Options Valuations Inc to compute the value of our option. The first step in the 
Super Lattice Solver 3.0 involves the creation of the lattice evolution of the underlying 
asset. What the model essentially does is multiply the up and down factors by the present 
value of the underlying asset at time zero to create the binomial lattice. Therefore we 
multiply the underlying value of the asset ($305 million) by 1.000536800340363 and 
0.999463487659644 to get the up and down factors respectively of the lattice. This is 
depicted in Figure 37. 
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Figure 37. Underlying Asset Lattice 


The second step in the Super Lattice Solver 3.0 is to create the option valuation 
lattice (Figure 38) using the values computed in the lattice of the underlying asset to 
generate the value of the option. 
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This requires two steps, the valuation of the terminal node and the valuation of the 
intermediate nodes using a process called backward induction [14], 

Option Valuation Lattice 
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Figure 38. Option Valuation Lattice 


The final node in the lattice represents the expiration of the option and the option 
value is simply the intrinsic or exercise value and represents the fair price of the Real 
Options with all the nodes leading up to the final node representing the value at a 
particular point in time given the evolution in the value of the software to that point. The 
terminal values in our lattice are the computed values that occur at maturity, while the 
intermediate values in the lattice are the computations that occurs at all periods leading 
up to maturity and. All these values are computed using backward induction. 

This value basically represents the value of the option if it were to be held as 
opposed to exercised at that point. Now it is worth noting the differences between the 
different types of call options. With a European option, there is no option of early 
exercise, and the binomial value applies at all nodes, for an American option, since the 
option may either be held or exercised prior to expiry, the value at each node is: Max 
(Binomial Value, Exercise Value), and for a Bermudan option, the value at nodes where 
early exercise is allowed is: Max (Binomial Value, Exercise Value); at nodes where early 
exercise is not allowed, only the binomial value applies. 
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Therefore given the values we obtained above, the binomial option valuation 
eomes out to be $251.63 million. Since the NPV of the investment is $250 million, our 
total option value (premium) in this case is $1.63 million. 

J. REAL OPTIONS ANALYSIS USING MONTE CARLO SIMULATION 

The final step in our methodology is to conduct a RO analysis in order to value 
the payoffs associated with our RO. This analysis involves the modeling of the value of 
the underlying asset which includes the option premium. We follow the steps outlined 
above. However in this case the expected value of the software investment effort is 
determined from the mean of the value which is computed after several simulations and 
the standard deviation or the variance of the value can be used to value options on the 
project. 

K. REALIZED REAL OPTIONS FRAMEWORK 


All the work we have conducted this far has been aimed at developing the 
foundations on which we could now use to establish a RO Framework (Figure 39) for 
supporting the strategic decision making associated with software related capital 
investments. 


1) We begin this process by computing the base case present value of the 
software investment effort using the NPV formula we discussed earlier in 
this chapter once the operational needs have been identified. Since we are 
computing NPV at time = 0, we omit the option premium factor Q .Thus 
NPV computation equates as follows: 


NPV= Z 


(l + rY 


(we can choose to refine our asset value using a Monte Carlo simulation) 

2) We then proceed to identify the uncertainties associated with the software 
investment effort by utilizing our proposed “2T” approach of uncertainty 
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elicitation from Chapter 3 and feeding these uncertainties into a risk 
model. The steps associated with the “2T” approach are as follows: 

a) Identify pragmatic uncertainties and associated input drivers. 

b) Identify Heisenberg-type and Godel-like uncertainties and 
associated input drivers 

c) Quantify and feed the input drivers of both types of uncertainties 
into the Risk Simulator provided by Real Options Valuations Inc. 

3) We then proceed to measure the risks posed by these uncertainties in the 
form of volatility by utilizing the Real Options Inc risk simulator in order 
to model the impact of the risks posed by the uncertainties. 

4) To “fit” our data, we attempt to identify sources of information that we 
could use to better understand these uncertainties and the risks they 
present. Specifically we look at both historical data and utilize the Delphi 
Method. In the case where historical data is not available, we rely on the 
Delphi Method or any other applicable proxies. 

5) Once we have gathered a better understanding of the uncertainties based 
on the techniques employed in Step 4, we proceed to quantify these 
uncertainties based on the risk they pose in order to estimate the volatility 
of the software investment program. 

6) We then solicit additional evidence from different independent sources 
and employ the Dempster-Shafer Theory of Evidence utilizing belief 
assignments and the Dempster’s combination rule to combine the evidence 
we obtained from our different sources. 

7) We utilize the evidence we obtained using Dempster-Shafer Theory to 
modify or refine our initial probability estimates obtained from historical 
data and obtain a refined volatility estimate of our investment effort. 

8) After completing all these steps, we are now ready to conduct our analysis. 
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9) We begin our analysis of the risks by proceeding to formulate the 
appropriate Real Options to hedge against the risks and develop a strategy 
tree depicting the strategic pathways. 

10) We value the options created in Step 8 using the Binomial Lattice 
approach and determine our option premium. 

11) We add this value of the option premium to our investment costs to 
compute the new asset value using 

NPV=J: -I+Q 

(\+ry 

We conduct a RO analysis by using random variations in a Monte Carlo 

simulation to model different scenarios as applicable 
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Figure 39. Realized Real Options Framework 


Now that we have established our real options framework we attempt to validate 
our approaeh to determine its fit to software-related capital investments. 
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VI. VALIDATING THE REAL OPTIONS ERAMEWORK 


Cost is the value of the foregone option.... Unknown 

In an attempt to validate our proposed approaeh, we deeided to apply the 
framework to the software eomponent (Future Combat Systems Network) of U.S. Army 
Future Combat System (FCS). The deeision to seleet this ease study as a validation 
meehanism was based on the reeent nature of the projeet, the high-risks assoeiated with 
software development due to the advaneed teehnologies involved, the ehallenge of 
networking all of the FCS subsystems together so that FCS-equipped units ean funetion 
as intended and the assoeiated outeome had a Real Options approaeh been applied. The 
assoeiated key eontributions of this ehapter are as follows: 

A. KEY CONTRIBUTIONS 

What we show in this ehapter using our Real Options methodology is how the 
deeision maker ean develop a series of “Call Options” whieh ean be ineorporated into the 
Future Combat Systems Network aequisition eontraet that would give the deeision-maker 
the right but not the obligation to exereise the options when the need arises. Essentially 
what the deeision maker would be doing by embraeing our methodology is identifying 
and paying for risk up-front at today’s priee and taking advantage of eost savings 
dependent on if the Call Option is “in-the-money” on the upside that the risks whieh the 
Real Options were ereated to address presents itself, otherwise the deeision maker forfeits 
the option. Specifieally these eost saving would be realized on “in-the-money” Call 
Options in the form of savings on differenees in prevailing market labor rates aetor in 
software development), sinee at this point in time, the exereise priee would be less than 
what was paid for the Call Option. We start by providing an overview of the Future 
Combat Systems program, the ehallenges faeing the Future Combat Systems Network 
and the applieation of our methodology within the eonstraints of two assumptions we 
make during the applieation of methodology. 
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B. FUTURE COMBAT SYSTEM (ECS) OVERVIEW 


The concept of the U.S. Army Future Combat System (FCS) was first introduced 
in October 1999, by then Chief of Staff of the Army (CSA) General Eric Shinseki in an 
attempt to convert all of the Army’s divisions (called Legacy Forces) into new 
organizations called the Objective Force with the intent of making the Army lighter, more 
modular, and most importantly more deployable [57]. These strategic considerations, 
were in line with enhancing U.S. Military operations and allowing US dominance in 
future conflicts across a full spectrum of threats in a global context. As part of this 
transformation, the Army adopted the Future Combat System (FCS) as a major 
acquisition program to equip the Objective Force [57]. Figure 40 below offers a visual 
depiction of the FCS. This transformation, due to its complexity and uncertainty, was 
scheduled to take place over the course of three decades, with the first FCS-equipped 
objective force unit reportedly becoming operational in 2011 and the entire force 
transformed by 2032. 



Figure 40. FCS Core Systems (From: [58]). Core systems of the Future 
Combat Systems depicting the software component (Future Combat 
Systems Network) as the “heart” of the overall program. 

The vision for the FCS was such that it would consist of four major components: 
Manned Ground Vehicles (consisting of 8 platforms). Unmanned Systems (which 
includes the Unmanned Aerial Vehicles (UAV), Unattended Systems, and Unmanned 
Ground Vehicles (UGV)), FCS Network (the software platform that provides the 
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communication and automation that creates the battle command environment), and 
Soldiers (who are ultimately empowered with the use of robotics and technological 
advantages). 

The Future Combat System (FCS) when deployed is intended to replace such 
current systems as the M-1 Abrams ta nk and the M-2 Bradley infantry fighting vehicle by 
tying together what was initially expected to be 18 manned and unmanned systems via an 
extensive communications and information network. However, over the last few years, 
this program has been plagued by program development issues with some technologies 
advaneing quieker than anticipated, others progressing along predicted lines, while still 
others experieneing significant delays, and skyrocketing eosts, with initial estimates of 
the FCS program jumping first from $91.4 billion in 2003 to $127 billion, and now to 
over $160 billion [63]. Coupled with the skyroeketing eosts, comes the issue of 
requirements inerease with souree lines of code (SLOC) almost doubling from an 
estimate 33.7 million SLOC initially, to 64 million SLOC and more recently tripling to 
95 million SLOC. While those costs have grown and grown, the list of equipment the 
FCS is actually supposed to deliver has been getting shorter, with the number of systems 
dropping from 18 to 14. Table 10 shows FCS code measured in source lines of code 
(SLOC) from inception until 2007. 



Original Estimate 

Estimate as of 

Estimate as of 

Percentage 


(May 2003) 

Jan 2006 

August 2007 

Increase 

SLOC 

33.7 million 

63.8 million 

95.1 million 

182 


Table 10. FCS Software Growth Estimates. 


C. SOFTWARE COMPONENT: FCS NETWORK 

Currently, there are many radio and computer systems using various different 
software which makes it difficult to communicate during both peacetime and combat 
operations. The intent of the FCS Network is to overcome this issue by enabling leaders 
at all levels to see first, understand first, act first, and finish decisively by eonnecting FCS 
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platforms to the Soldier at every eehelon, from Brigade to Squad while at the same time 
giving the ability to integrate eommunieations with other Department of Defense 
ageneies and US allies [58]. The software eomponent of the PCS program is intended to 
provide a eommunieation platform for soldiers to eommunieate through a wireless 
network in near real-time with multiple other “assets” sueh as hovering drones, remotely 
eontrol robots to defuse bombs, firing laser-guided missiles at enemies on the move, 
eondueting a video teleoonferenee in a tank rumbling about 40 mph in the haze of battle 
utilizing the same network at the same time. 

The design is based on ensuring the availability of the eommunieation network at 
all times so that even if a soldier should lose their eonneetion, the software would 
automatieally “eorreet itself,” retrieving the information within seeonds without 
rebooting by finding and utilizing an effieient mathematieal algorithm to reeonneet the 
soldier. The PCS Network has its own operating system known as the System-of-Systems 
Common Operating Environment with over 100 interfaees or software eonneetions to 
systems outside PCS. The PCS Network is made up of 5 layers: Sensor/Platform Layer, 
Applieation Layer, Serviees Layer, Transport Layer, and Standards Layer. These layers 
provide diversity in waveform, frequeney and environment to ensure there are multiple 
paths to transport the data. Eaeh network is tailored to support the speeifie needs of the 
end users. Depending upon the eommunieation eonfiguration most users will be provided 
with multiple layers of aeeess. The ECS software is eurrently being developed in four 
diserete stages, or bloeks, with eaeh bloek adding ineremental funetionality spanning 
approximately eight funetional areas (eommand and eontrol, simulation, logisties, 
training, manned ground vehieles, unmanned aerial vehieles, unmanned ground vehicles, 
and warfighting systems) [64]. 

D. BENEFITS OF ECS 

Besides the operational benefits in terms of tactical advantages and increased 
situational awareness, cost savings are expected to be realized from the ECS program 
because the new combat vehicles in the ECS are designed to share common hulls and 80 
percent common parts. This would manifest into fewer spare parts and streamlined 
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training and functions of mechanics; instead of needing speeialists for a mixed fleet. 
Furthermore, the redueed weight of the eombat vehieles also translate into redueed airlift 
requirements in terms of eapaeity (Air Foree Cargo airerafts would be able to earry more 
eombat vehieles due to redueed weight) in the ease of operational deployments, whieh 
also translate to reduee wear and tear on Air Foree aireraft and fuel requirements. While 
the FCS Network is just one eomponent of the overall program, it is the underlying 
platform on whieh the sueeess of the FCS program depends, henee the need to apply a 
diseiplined approaeh to its aequisition and development. Before identifying the 
ehallenges faeing the FCS Network from both a managerial and teehnieal perspeetive, we 
first we state our key assumptions. 

E. ASSUMPTIONS 

Due to data limitations at the level of granularity at whieh we would have hoped 
we made the following two assumptions whieh we justify later on in this ehapter at the 
applieable situation. 

1) We estimated the future benefits of the Future Combat Systems Network 
(Asset Value) under traditional NPV and assumed it was positive (i.e., 
benefits outweigh costs) and, further later on in the study, transposed the 
eosts of the overall FCS program by making it the eost of the FCS 
Network (software eomponent) sinee detailed breakdown of cost data was 
not available for eomparables. 

2) We assume the independent assessments provided by the Cost Analysis 
Improvement Group (CAIG) and the Institute of Defense Analysis (IDA) 
inelude belief assignments. In other words, we assume that an exeeutive, 
during the deeision making proeess, is provided not only with a raw set of 
the risk faetors of requirements ereep, integration risk, performanee risk, 
but with additional measures: belief in the estimation of the risk faetors 
and eertainty of the estimation. 
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F. 


TECHNICAL CHALLENGES 


To date, the software development eomponent of the PCS Network represents the 
largest software development effort in DoD history with a eurrent projection of 95.1 
million SLOC providing 95% of the PCS functionality. Prom the onset, its development 
has been plagued by technology maturation issues since most of the capabilities were 
conceptual in nature and needed to be demonstrated so that the associated software 
requirements developed. Purthermore, while the concepts of the PCS does have its 
operational merits, it has been compounded by requirements challenges in which meeting 
some requirements has the unintended consequence of working against another 
requirement [63]. 

It is also currently envisioned that the 95.1 million lines of software code of the 
PCS program would be based on three categories (Pigure 41) new code, reused code, and 
commercial-off-the-shelf code (COTS). Given the huge spike in the Source Lines of 
Code (SLOC) estimates from 33.7 million in 2003 to today’s estimate of 95.1 million, 
there is a high probability that the number of lines of code required for the program 
would increase as the Army learns more about the technology and its design concepts. 


New 



■ New □ Resused h Commercial of the Shelf (COTS) 


Figure 41. FCS Projected Software Lines of Code (in thousands) (From: [58]). 
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Furthermore, while new software code represents the greatest challenge of all the 
three categories of code due to the fact that it has to be written from scratch, it is also 
highly speculative that the amount of software code that could either be reused or adapted 
(COTS) is inaccurate. Hence these estimates which might be somewhat conservative and 
could easily translate to greater time and effort to develop software than is currently 
planned therefore resulting in cost overruns. 


Currently the FCS warfighter operational requirements stand at 544 which 
translates to approximately 11,697 system-of-systems requirements. Of the system-of- 
system requirements, 289 have “to be determined” items, 819 have open issues to be 
resolved individual system level requirements, resulting in 51,944 system level 
requirements [58]. The system level requirements provide the specificity needed for the 
contractors to fully develop detailed designs for their individual systems. Figure 42 below 
illustrates how the FCS requirements are translated from the warfighter to the individual 


systems. 



Figure 42. 


Flow of FCS Overarching Requirements to System-Level 
Requirements (From: [58]). 


The development strategy of the FCS Network is based on the evolutionary 
approach as can be seen in Figure 43 in which the practice is to overlap builds more than 
the traditional spiral model does, by beginning the requirements phase of the next build 
before the testing phase is completed on the previous build so that the next build is ready 
for design by the time the former build has completed testing. 
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Build 0 



Figure 43. FCS Spiral Development Strategy and Software Life Cycle 

Reviews (From: [64]). 


However, in the midst of evolving requirements, this approach has caused 
developers to interpret and implement changes in requirements well into the design and 
code phases, compromising the amount of time allotted for testing. This is not to say that 
the requirements should have been defined more quickly, rather, the requirements 
instability is amplified by the relative immaturity of the program, coupled with its 
aggressive pace, the pronounced overlap of the FCS builds and the cascading effect on 
software development [64], 

G. MANAGEMENT CHALLENGES 

Program costs and schedule concerns driven by technical risks and uncertainty 
form the basis of the managerial challenges. From an original modest cost of $92 billion 
for the whole program for all 18 systems, the costs of the FCS has skyrocketed to over 
$163.7 billion dollars for only 14 of the 18 systems according to Army estimates [64]. 
Based on the current development and delivery schedule, about 10% of the software 
which is considered to be the most difficult part of the development effort by program 
officials is planned to be delivered and tested after the early 2013 production decision 
which would limit the knowledge available to decision makers at that point [64]. 
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Block 

Percentage of total Software completed 

Delivery Date 

0 

5 

Sep-05 

1 

30 

Dec-07 

2 

61 

May-10 

3 

90 

Oct-11 

4 

100 

Oct-13 


Table 11. FCS Software Blocks, Percentage of Completion, and Delivery Dates (From: [64]). 

Currently, the greatest risks faced by the program, besides the unknown total costs 
of the system, stems from concerns that the overall FCS would not deliver the required 
capability within estimated resources [61]. Compounding these problems is the lack of a 
top-level requirements and architecture definition which in effect leaves system-level 
requirements incomplete until the preliminary design review in 2009, which further 
affects the accuracy of projected SLOC. Therefore given all the challenges facing the 
software component of the FCS program (FCS Network), we summarize our key finding 
of sources of uncertainty of the FCS Network based on the current challenges. 

H. MANAGERIAL UNCERTAINTIES 

As mentioned earlier and depicted in our “2T” model, constraints of people, time, 
functionality, budget, and resources form the basis of the uncertainties faced the program 
manager. Therefore using this paradigm, based on the information we have gathered thus 
far, we can frame managerial uncertainties along two lines: Estimation (size and costs) 
Uncertainty and Schedule Uncertainty. We discuss this further. 

I. Estimation Uncertainty 

Generally speaking, poor size estimation is one of the main reasons that major 
software-intensive acquisition programs fail. The overall costs of the FCS Network 
program continue to be plagued by estimation difficulties associated with changing 
requirements, immature architecture, with the U.S. Army, the Institute of Defense 
Analysis and DoD’s Cost Analysis Improvement Group (CAIG) reporting independent 
and different cost estimates of $163.7 billion, $166.7 and a range of $203 - $234 billion 
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respectively. The difficulties associated with accurate software estimating is an indication 
that complexity increases as the design is better understood, resulting in the increases in 
the level of effort and possible increases in the development schedule, and ultimately 
resulting in increased costs. The general consensus as reported in reports authored by the 
Government Accountability Office (GAO) is that the Army’s estimates are limited by the 
low level of knowledge in the PCS program today since the Army does not have a base of 
mature technologies and well-defined system-level requirements. Therefore, the Army 
resorts to making significant assumptions about how knowledge will develop [58]. Table 
12 below highlights the trend of cost growth from the inception of the program. 



Independent 

Army Original Estimate Current Estimate Cost Estimate 

Base year 2003 dollars 
Total 

May-03 Dec-06 May-06 

$91.4 $163.7 $203.3 - $233.9 


Table 12. Comparison of the Original Cost Estimate and Recent Cost Estimates 
for the ECS Program (in billions of dollars) (Erom: [64]). 


2. Scheduling Uncertainty 

In comparison to the Joint Strike Fighter (JSF) which has only approximately 24 
million SLOC [64] with a delivery timeline of 12 years using a similar 5 block 
incremental delivery approach, we believe that the FCS Network development schedule 
to be too risky as the software development and testing schedules appear to be based on 
the paradigm of “development success”, with have little margin to accommodate delays. 
Furthermore, in light of the uncertainty surrounding the estimation of the size of the 
software, we believe that the schedule does not necessarily or adequately reflect the 
potential impact that uncertainty in size could have on the acquisition schedule. 
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I. 


TECHNICAL UNCERTAINTIES 


There is a preponderance of technical uncertainties surrounding the development 
of the PCS Network by virtue of its reliance on the successful development of conceptual 
critical technologies. For example, technological maturation issues such as the limitations 
associated with wirelessly transmitting still images, video and audio have plagued the 
program. Using our “2T” paradigm we frame technical uncertainties along the following 
categories: Requirements Uncertainty, Integration Uncertainty, and Performance 
uncertainty. We expand on these uncertainties below. 

1. Requirements Uncertainty 

The lack of adequately defined requirements (Table 13) is one of the leading 
problems in the software development effort. Without adequate definition and validation 
of requirements and design, software engineers could be coding to an incorrect design, 
resulting in missing functionality and errors. This problem is further compounded by the 
lack of top-level requirements and architecture definition which in effect leaves system- 
level requirements incomplete until the preliminary design review in 2009, further 
affecting the accuracy of projected lines of code. 



Pooriy 

defined 

requirements 

Late 

requirements 

Missing 

requirements 

Software Partitions 

X 


X 

Combat identification 

X 

X 

X 

Battie Command and Mission Execution 

X 

X 

X 

Network Management System 

X 

X 

X 

Smaii Unmanned Ground Vehicie 

X 

X 

X 

Training Common Components 

System of Systems Common Operating 

X 

X 

X 

Environment 

X 

X 



Table 13. Software packages and associated requirements problems (From: [59]). 
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2 . 


Integration Uncertainty 


There are over 100 software vendors involved in the development of software 
programs for PCS, including 14 first-tier contractors, and many other sub-contractors 
[59]. Due to the amount of contractors involved, there is a lot of uncertainty surrounding 
how successful the different components would integrate due to the amount of 
collaboration that the effort would involve. This issue is further compounded by the 
inherent competition amongst software suppliers and their unwillingness or inability to 
share information and rapidly negotiate changes in products and interfaces, which could 
lead to missed delivery schedules, significantly reduced operational capabilities, and less 
dependable system performance. 

3. Performance Uncertainty 

Requirements changes, integration issues and late testing pose the risk of that the 
PCS Network might not yield fully functional software that performs as desired due to 
the complexity and functional scope. Purthermore, security is also a major concern from 
an operational perspective, with worries about the possibility of hackers implanting 
viruses and malicious code, thereby increasing the possibility of software failure when 
fielded in an operational mode. 

Based on the uncertainties we have identified thus far, we can summarize the risks 
affecting the value of the PCS Network as being; 

1. Requirements Creep 

2. Estimation Accuracy (size and cost of the software) 

3. Performance Risk 

4. Software Integration 

With this information in hand we now searched for historical data on a similar 
software development (size and scope) effort as the PCS Network that exhibits the same 
risk factors as the PCS Network effort. In this case we chose to select the Joint Strike 
Pighter program (JSP). 
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J. 


BASIS FOR SELECTING THE JOINT STRIKE FIGHTER PROGRAM 


In our examination of historical data, we were able to determine that the historical 
data presented by the JSF program closely reflected the risks and challenges faced by the 
FCS program. Hence we elected to compare the JSF program to the FCS program for the 
following reasons. 


I. JSF Technology Maturation Risks 


Requirements instability perpetuated by technology maturation issues plagued 
the JSF program from the onset just like the FCS program, with only two of the JSF’s 
eight critical technologies being fully mature and three other technologies being 
immature even after design review had been completed. This is a similar situation to the 
FCS program where only two of the program’s 44 technologies are fully mature and 30 
are nearing full maturity. Maturing critical technologies during development led to cost 
growth in the JSF program and is also the case in the FCS program. 


Attainment of Product Knowledge 



(5/03) (1.'08) review (2/13) 

( 2 / 11 ) 

Figure 44. FCS Program Management (From: [62]). 
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Attainment of Product Knowledge 



f6/071 


Figure 45. JSF Program Management (From: [62]). 


2. JSF Program Management (Cost Risks) 

The JSF and FCS contracts were both awarded on a cost reimbursable basis for 12 
years. These contract vehicles allowed payments to the contractors on the basis of time 
spent on the project at pre-determined man-hour rates rather than a firm fixed price. 
Hence, the risk for all cost growth over the estimated value rests with the DoD. As of 
fiscal year 2007, DoD anticipates having to reimburse the prime contractors on these two 
programs nearly $13 billion more for their work activities than initially expected [62]. 

3. JSF Software Size (SLOC) 

The JSF program represents the second largest software-intensive program behind 
the FCS. Figure 46 and Table 14 below highlight the SLOC, and cost and development 
schedule for the JSF. 
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Figure 46. SLOC of Historical Acquisition programs of 
Software-Intensive-Weapons Systems. 



Program 

inception 

November 

1996 

System 
deveiopment 
start (October 
2001) 

Program 

Rebaseiine 

(December 

2003) 

Data as of 
December 

2005 

Cost Estimates 
(Then Year $ in 
biiiions) 

Deveiopment 

$24.8 

$34.4 

$44.8 

$44.5 

Estimated Deiivery 
Date 

First Operationai 
Aircraft deiivery 

2007 

2008 

2009 

2009 


Table 14. Joint Strike Fighter Cost and Schedule Increases from program inception. 


Our goals and objectives in utilizing the JSF historical data is to investigate, 
analyze and incorporate any lessons learned in the JSF program since development of the 
JSF began prior to the FCS. 

K. RISKS IMPACT ON FCS NETWORK 


In software development, requirements instability have been found to have a 
profound impact on a program’s schedule and drive up costs due to Research, 
Development, Test & Evaluation (RDT&E) costs increases associated with the 
requirements changes. Therefore, given the situation with the ECS Network, we can 
depict the relationship between the risks as follows. 
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Figure 47. Impact of Risk on FCS Network. 


Given the impact that requirements instability has on costs, we attempt to 
determine the rate of change of requirements or the volatility of requirements. To 
accomplish this, we utilize the Caper Jones’ approach which is a transposition from the 
financial industry. Jones asserts that the method of average percentage of change of the 
overall requirements volume lacks information, because it does not give any information 
on the time in which the change occurred, a key factor that is important to determine in 
software engineering, since requirements changes become more expensive to implement, 
the farther we are into the software development process. 

Jones therefore uses the compound monthly requirements volatility rate [67] to 
express the time aspect. Calculating monthly requirements volatility rates, as defined by 
Jones, is a transposition from the financial world. The time value or future value of 
money is well-known in the field of accounting as compound interest or CAGR, short for 
compound annual growth rate. By transposing from compound growth rate in finance we 
assume that requirements are compounded within a project [67]. The basic financial 
equation is given as follows: 
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r = 


SizeAtEnd 

SizeAtStart 


\ 

1 xlOO 


which translates to 


r = 


SLOCAtEnd 

SLOCAtStart 


\ 

1 xlOO 


where ^is the time period in years during which the estimates were observed 


However, SLOC is not a suitable proxy for measuring requirements volatility 
because more often than not, it is dependent on the type of programming language being 
used and also does not take COTS into consideration, which represents a sizeable portion 
of the PCS program. Ideally we would have preferred to use an alternative proxy such as 
Function Points, which is a better metric for the size of the software requirements 
irrespective of how the software will be developed. However due to lack of data to 
determine the number of Function Points associated with the FCS Network program, we 
resorted to use SLOC to demonstrate our approach in determining requirements volatility. 
Note that the same approach can be used to determine requirements volatility using 
Function Points if we had information on the number of Function Points in the FCS 
program at the time of our study. 

We compute the requirements volatility of both the FCS Network and the 
software component of the JSF using the SLOC data, and summarize the information we 
have thus far in Table 15, and compute the volatility of requirements using the formula 
above. 


Program (Period) 

Beginning SLOC (Miiiions) Ending SLOC (Miiiions) 

FCS (2003 - 2008) 
JSF (2001 - 2008) 

33.7 95.1 

17.2* 22.9 


Table 15. Comparison between JSF and FCS SLOC 

*We estimate the initial SLOC of JSF program by utilizing the average of 25% increase 
in requirements reported by GAO in their assessment of selected weapons systems [62] 
and determine the initial SLOC estimate of the JSF to be 17.2 million SLOC 
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Program 

Beginning SLOC (Miiiions) 

Ending SLOC (Miiiions) 

r (Voiatiiity) 

FCS (2003 - 2008) 

33.7 

95.1 

12.37754868 

0.282516275 

JSF (2001 - 2006) 

17.2 

19* 



Table 16. Volatility computation using Caper Jones approach. 

*We utilize the SLOC of 19 million as reported in GAO assessments in 1996. We utilize 
this value beeause the PCS Network growth reported in this table is based on a 5 year 
period of SLOC growth, hence our need to size the JSF program accordingly as it 
represents a 5 year period of SLOC growth 


We further estimate that over the 5 year period of observation depicted in Table 
15, the PCS Network was 5.0 times the size of the JSP program (i.e. 95.1/19). Therefore, 
to adequately size the JSF program, we multiply its current volatility by 5 and get a 
resulting volatility estimate of requirements of 1.440833 

In comparison with the averages reported by Jones’ research on industrial 
averages (Table 17), we can see that requirements volatility in the PCS Network exceeds 
traditional averages of military software, making it a risky venture that is approaching the 


“out of control” category. 


Software Type 

Average 
voiatiiity (%) 

Maximum voiatiiity 
(%) 

Out of control (%) 

MiS Software 

1.2 

5.1 

>5 

Systems Software 

2 

4.6 

> 5 

Miiitary software 

2 

4.5 

> 15 

Commerciai software 

2.5 

6 

No info 

Civiiian government software 

2.5 

No Info 

No info 

Web-based software 

12 

No Info 

No info 


Table 17. Caper Jones’ industry averages and maximum rates in various 

industries (From: [67]). 

Since the SLOC estimates of the FCS program are nearly 5 times that of the JSF 
program, we extrapolate the JSF data to mirror that of the FCS program using our Risk 
Simulator modeling toolkit. We first determine the value of the FCS Network as after all, 
the intent of running a Monte Carlo simulation is to determine the volatility of the returns 
on the FCS Network based on the modeled risks. 


134 








L. VALUATION OF THE FUTURE COMBAT SYSTEMS NETWORK 

In order to value the FCS Network, we need to use a finaneial model and the 
development of the finaneial model requires that we either have or eompute the 
following four faetors. 

1. Costs of developing the Future Combat Systems Network 

2. Quantifiable benefits 

3. Diseount Rate 

4. Time period 

As of 2008, the overall FCS program eost was estimated at $163.7 billion [59], 
with the software eomponent alone providing provides 95% of the FCS funetionality 
[65]. Furthermore determining the diseount rate and the time period is quite simple as this 
information is readily available. For diseount rate, we use the spot rate on the U.S. 
Treasury bill and a time period of 10 years refieets the software development/delivery 
sehedule. 

However quantifying the benefits of the FCS Network in terms of a monetary 
value is a very eomplieated task. While the U.S. Army provides guidanee on quantifying 
benefits based on faetors sueh as eost savings, eost avoidanees and produetivity 
improvements, the numerie benefits eannot be easily estimated. Even though the overall 
FCS program would feature affordable sustainment eosts, redueed logisties requirements, 
and a deerease in erew size as eompared to the eurrent systems as gathered from the 
initial justifieation for investment in the FCS program, we did not have suffieient data 
that refleeted these benefits in the form of eost savings or eost reduetion at the time of our 
study. To eompound this, it is very diffieult to proportionately alloeate the eost savings to 
the FCS Network. Although the FCS Network is a eritieal eomponent of the overall FCS 
program, these eost savings eannot be alloeated to the FCS Network alone beeause the 
FCS Network is just a teehnology platform providing the underlying funetionality of the 
overall FCS program, with many other eomponents sueh as lighter manned vehieles 
eontributing to the eost savings. 
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Consequently we make an assumption, the first of two assumption we make 
during the validation of our approaeh. 

1. Assumption 1 

We assume that the benefits eould far exeeed the eosts under traditional NPV 
analysis or with the goal is to manage the development (investment) eosts within the 
eonstraints of the given budget without redueing required funetionality. 

2. Justification 

We believe this to be a reasonable assumption beeause of the pereeived benefits 
obtained over an extended time period in the form of eost savings due to the replaeement 
of legaey systems, vehieles and the taetieal advantages the troops on the battle field 
would enjoy. 

We take the approaeh that the regardless of the future benefits provided by the 
PCS Network, the value is derived based on how mueh we ean sueeessfully manage the 
software development or investment eosts so that we ean maximize the returns of the 
investment by keeping eosts low via suoeessfully planning and paying a risk premium up 
front for risks. In other words, we take an investment eost management approaeh towards 
maximizing future returns. 

Sinee the government model is to provided the needed eapability at least eost to 
the taxpayer, our foeus on “managing” the investment eosts to make sure that we do not 
overrun the eosts and therefore maximize returns. Furthermore in aetuality, sinee the 
program is eurrently underway, we ean say with 100% eertainty that a finaneial model, a 
pre-eondition for the applieation for Real Options, was developed before the 
eommeneement of the aequisition effort. 
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Based on this underlying assumption we claim that under traditional NPV 
analysis, the NPV of the PCS Network using our formula below is positive. 


NPV= Z 


(1 + r)^ 


I is the (initial) development cost and represents 
T is the (initial) development time or time to market, 

C is the asset value and represents the total value of the positive cash flows that the 
project is expected to generate during its lifetime, 

M is the operation cost and is the total value of all negative cash flows of the operation 
phase, calculated at time T. 

(C -M) is always positive 

r is the rate at which all future cash flows are to be discounted (the discount rate). 


3. Assumption 2 

For cost estimation purposes we assume that the overall PCS program is software, 
therefore we use the costs of the overall PCS program in our computation. 

Thus based on our assumptions, we develop our financial model to mirror a 
positive NPV based on the following key input factors 

I = $163. 7 billion 

T = 13 years 

r = 3.0% 

C - M = $10 trillion 

Since we did not have detailed data, we assume an asset value of $10 trillion and 
operating cost of $0. Based on these values, we are able to compute the NPV of the PCS 
Network as being approximately $6.4 trillion as shown in Table 18. 
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MPV oH Future Fieturns on a 

Investment of ^163.7 billion in an 
Asset Valued at ^10 Trillion 


FCS 

$3,401,505,349,050.37 



Table 18. NPV of FCS Network. 

4. Rationale for Assumption 

We assume these cost estimates reflect the estimates of the FCS Network as 
opposed to the overall FCS program for the purposes of our study since we cannot clearly 
delineate between software costs and hardware costs from the inception of the program. 
We therefore treat the overall FCS program as consisting of “software only” for cost 
purposes and assume the overall costs of the FCS Network is the cost of the overall FCS 
program. By doing this we can then develop a model based on the following; 

Value of Army’s overall FCS cost estimates = Army’s FCS Network estimate 

Value of CAIG’s overall FCS cost estimates = CAIG’s FCS Network estimate 

Value of IDA’s overall FCS cost estimates = IDA’s FCS Network estimate 

We made these assumptions because at the time of our study, we did not have 
access to the detailed data of both IDA’s and CAIG’s estimates of only the FCS Network. 
The data we reviewed at the time of our study did not provide information at the level of 
detail that was beneficial to our study, consequently we were not able to isolate the FCS 
Network (software component) from the overall FCS program based on their independent 
CAIG’s assessment. Therefore, for the purposes of validating our framework, we assign 
the values of the overall FCS program provided by the Army, IDA and CAIG as our 
software development/investment costs. In other words we now assume that the estimates 
provided by the Army, CAIG and IDA of $163.7 billion, $218.5 billion (average of range 
provided) and $166.7 billion respectively represent the costs of the FCS Network 
investment. 
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Now that we have addressed the issue of asset valuation, we proeeed to model our 
risks and determine the volatility of the returns on the investment using our modeling 
toolkit. 


M. VOLATILITY DETERMINATION 


Based on the uneertainties we have identified thus far, we now feed risks into our 
risk model in the Risk Simulator (Table 19) to observe the impaet on the value of the 
future returns on the overall PCS program using the JSF program as a proxy in our data 
fitting. Based on our eomputation volatility of eaeh of the risk faetor we identified using 
the Caper Jones’ approaeh, we are able to develop probability estimates, and run a Monte 
Carlo simulation. Sinee we are dealing with random quantities whieh are positive in 
nature and whose values eannot fall below zero we seleet the lognormal distribution 
whieh utilizes the following four mathematieal eonstruets to eompute four key measures 
known as “moments” (returns, risk, skewness and kurtosis) of our logarithm distribution 
These measures are eomputed using the following formulas as outlined in ehapter IV of 
this study. 

[ln(x) - ln(//)]^ 

f(x) = — e 2[ln(cr)]2 x > 0; // > 0, cr > 0 

x->l7.n\xv{G) 


mean = exp 


V ^ J 


standard deviation = -^exp(cr^ + 2//)[exp(cr^)-1] 


skewness = 


-Jexp(cr^) -1_[2 + exp(cr' 


)) 


exeess kurtosis = exp (4cr^) + 2 exp(3cr^) + 3 exp(2cr^) - 6 
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The model is run and the simulation ealeulates numerous seenarios of the model 
by repeatedly picking values from the lognormal distribution for the uncertain variables 
using the values in the model. 




RiskFjclois 



PolFutmRdwsoy 
InvesInienlDl lISlFHoniiiin 
AsHti/sWatSIOTiilta 

Coiient 

Rtflimfnt: PtatJScfctJule 
(SLOC) (Hois) RlamfJ Costs 

Refluitoifnts 

CwpRisk 

Inlaigialion 

Risk 

Schtdolt 

Owtm 

PfllOIIWIlM 

Risk 

SollwCosI 

Estimilm 

Unit Cost 

Hoil|Cost 

Finii Costs Due to Risk 

Fjctots 

FCS 

OTWOWH 

3500000) i «1,000,000,00 

lOOtt 0,587. « 

0,38'/! 31377 

81,733,15 8I,35WS66,67 8670,326,337,357,6f 


Table 19. Screen capture of risk model developed in the Risk Simulator. 



Type: Two-Tail, Lower: -Infinity. Upper: Infinity, Certainty: 100.0000% 

Figure 48. Risk Simulator output depicting returns on a 
$163.7 billion Investment in the FCS Network. The frequency histogram shows the 
frequency counts of values occurring in the total number of trials simulated. The 
vertical bars shows the frequency of a particular NPV occurring out of the total 
number of trials, while the cumulative frequency depicted by the smooth line shows 
the total probability of all values at and below the maximum NPV occurring in the 

forecast. 
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Histogram Statistic Preferences | Options | 


Statistics 

Result 


Number of Trials 

1000 


Mean 

5.756007E-H)12 


Median 

5.866101 E+012 


Standard Deviation 

4.986206E+011 


Variance 

2.486225E-H)23 


Coefficient of Variation 

0.0866 


Maximum 

6.472998E+012 


Minimum 

2.962358E-H)12 


Range 

3.510640E+012 


Skewness 

-1.3724 


Kurtosis 

2.8821 


25*4 Percentile 

5.502169E+012 


75*4 Percentile 

6.127237E+012 


Percentage Eiror Precision at 95*4 Confidence 

0.5369V. 



Figure 49. Volatility of the returns on the $163.7 billion Investment 
in the FCS Network. The forecast statistics summarizes the distribution of the forecast 
values in terms of the four moments of a distribution. 


In Figure 49, we observe that the volatility of the returns associated with the FCS 
Network investment, as indicated by the correlation of variation, has a value of 0.0866, 
and there is a decrease in the NPV from $6.4 trillion to $6.1 trillion, a direct reflection of 
the risk factors of requirements volatility, integration volatility, schedule overrun, 
performance risk and software cost estimation risk which we factored into our valuation 
of the FCS Network. 

We use coefficients of variation as a proxy of standard deviation because it is a 
relative measure of the standard deviation expressed as a percentage of the mean. Even 
though we have identified five possible risks as being the key contributors to the 
volatility of the returns of investing in the FCS Network, we model only the requirements 
risk and attempt to determine that, given the volatility of requirements, how does this 
impact the schedule and consequently affect the value returns of the FCS Network. 

Next we attempt to refine our volatility estimates using Dempster-Shafer Theory. 
Since a key proposition of Dempster-Shafer Theory is that all sources of information be 
independent, we satisfy this proposition by utilizing independent estimates provided by 
the Cost Analysis Improvement Group (CAIG) [64] and Institute for Defense Analysis 

(IDA) [59]. We review the background of both organizations in the next section. 
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N. BACKGROUND OF THE EXPERTS 


The Institute for Defense Analyses (IDA) is a non-profit organization first 
established in 1947 responsible for providing independent teehnieal analyses of weapons 
systems and programs. They administer three federally funded researeh and development 
eenters to provide objeetive analyses of national seeurity issues, partieularly those 
requiring seientifie and teehnieal expertise, and eonduet related researeh on other national 
ehallenges. The seeond independent entity, the Cost Analysis Improvement Group 
(CAIG) was formed at the behest of the Deputy Seeretary of the Department of Defense. 
In 2006, the deputy seeretary issued a poliey direetive (5000.04) direeting that any Major 
Defense Aequisition Program (MDAP) undertaken by the DoD undergoes an independent 
review and assessment. As an objeetive entity in Offiee of the Seeretary of Defense, the 
CAIG was identified as the independent body responsible for eondueting independent 
eost assessments of Major Defense Aequisition programs and serving as the prineipal 
advisor on matters of program life-eyele eost. 

O. REFINING VOLATILITY USING DEMPSTER SHAFER’S THEORY 

To eompute the volatility of FCS Network based on the risks identified above, we 
first establish the frame of diseernment 0 whieh eonsists of the set of mutually exelusive 
alternatives or possibilities these risks pose to the FCS Network. In the ease of CAIG, 
they provide a range of $203 - $234 billion for their estimate, thus we take the average to 
refleet the median ($218.5 billion) of their estimate. The IDA, on the other hand, provides 
an estimate that is $3 billion over the Army’s estimate. An important point worth 
mentioning is that the Army does not agree with both assessments as program offieials 
believe that the independent estimate of researeh and development eosts is too high 
beeause it is too eonservative regarding risks [64]. 

We summarize the independent estimates provided by both organizations in the 
Table 20 below. 
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Source of Assessment 

FCS Estimates 
(billions) 

Army 

$163.7 

Cost Analysis Improvement Group 

$218.5 

Institute of Defense Analysis 

$166.7 


Table 20. Summary of the three independent estimates of the FCS program 

The first step in application of DST is to establish belief functions. To accomplish 
this, we examined the primary differences between the CAIG estimate, IDA estimates 
and the Army’s estimates. CAIG’s estimate appeared to be the most conservative 
estimate. They determined that the FCS software development would require more time 
and effort to complete than the Army had estimated, resulting in several additional years 
and additional staffing beyond the Army’s estimate to achieve initial operational 
capability [64]. IDA found that Army plans for developing FCS, including the network, 
were optimistic with regard to time and money needed for the program and therefore 
projected at least $3 billion in additional FCS development costs due to unplanned 
software effort including code growth, software integration difficulties, and longer 
development schedules [59]. However based on our studies, it appeared the general 
consensus was that there was no question regarding the issue of requirements volatility. 
In other words, based on our assessment of GAO reports, we were led to believe that the 
probability of requirements creep was fairly consistent across all sources. Therefore, 
while the probability of schedule impacts might be debatable across all the different 
estimates, the probability of requirements creep was the same. 

We now establish the frame of discernment for both the CAIG’s and IDA’s 
assessment of the FCS Network based on the set of the risk factors affecting the 
investment as follows; 

0 = {SE, SR, SI, SS, FP} 

where 

Estimation Accuracy Risk (SE) - Under/overestimation risk 

Requirements Creep Risk (SR) 
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- Requirements volatility risk 





Software Integration Risk 

(SI) 

Delivery Risk 

(SS) 

Performance risk 

(EP) 


Integration risk 

Risk of schedule overruns 

Risk of software not meeting user needs 


CAIG Belief functions are as follows 3; 

fie/({SR}) = 0.90 

BeliiSR, SE}) = 0.90+ 0.08 
BeliiSR, SE, SI, SS, EP}) = I 

We assign fie/({SE}) the next highest probability assignment because the effects of 
requirements creep on the cost estimation of the ECS Network. 

The belief functions were based on the masses of evidence gathered as follows 

m({SR}) = 0.90 

m({SR, SE}) = 0.08 

m({SR, SE, SI, SS,EP}) = 0.02 


Similarly we come up with the following Belief assignments based on IDA’s 
assessment using the same analogy above. 

fie/({SR}) = 0.95 

BeliiSR, SE}) = 0.95+ 0.03 
fie/({SR, SE, SI, SS,PP})= I 

m({SR}) = 0.95 

m({SR, SE}) = 0.03 
m({SR, SE, SI, SS,EP}) = 0.02 


3 To determine CAIG’s subjective probability assignments on the Army’s estimate, we need to examine the mass of 
evidence that CAIG is able to gather to either validate or discredit the Army’s estimate. Since this information was not 
explicitly available at the time of this study, we resort to a different approach for illustrative purposes. 
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We now attempt to eombine the information we have so far. I.e., CAIG’s 
assessments and IDA’s assessment by eomputing the orthogonal sum in the form m = mi 
® m 2 where mi (CAIG) and m 2 (IDA) represent the basie probability on our frame of 
diseernment©. (Table 21) 



CD:lAn9l|:!i!linpiDveinenlGiDU|iEsliinjle|CAIG| 


CostlntfejseFeclois 

ISaSEImliOt |SR.SE,SI.SS,FP|ml:l.l2 

Insliluleol Defense 
AneljsispDAI 

{Slt|mM.3S 

{SI1,SE)mM.D3 

|SR,SE.SI,SS.FP|mZ:W 

D.t55 

11 

yi3 



out 

0.i 

yi 

D.il 


Table 21. Screen capture of orthogonal matrix. 

We obtain the following Evidenee Funetions 

nil © m 2 ({SR}) = 0.855 + 0.076 + 0.019 + 0.027 + 0.018 = 0.995 
mi © m 2 ({SR, SE}) = 0.0024 + 0.0006 + 0.0016 = 0.0046 
mi © m 2 ({SR, SE, SI, SS, FP }) = 0.0004 

To examine the degree of eonfliet we revisit Eqn 17 above depleted as; 

BnC=A 1 ^ 

where 

K= ^mi(B)m2(C) 

BoC=0 

However sinee we have no null interseetions, the sum of all sueh sets is in our matrix in 
Table 21 is equal to 0. Therefore \ - K = \ 

The plausibility (eertainty) of these beliefs are eomputed based on the following doubt 
funetions 

Dou({SR, SE, SI, SS, FP}) = Bel (0) = 0 
Dou({SR, SE}) = Bel (SI, SS, FP) = 0 
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Dou({SR }) = Bel (SE, SI, SS, FP) = 0 
Thus plausibility is as follows: 

Pli{SR}) = 1- Dou({SE, SI, SS, FP}) = I 

P/({SR, SE }) = I- Dou({SI, SS, FP}) = I 

R/({SR, SE, SI, SS, FP}) = I- Dou({SR, SE, SI, SS, FP}) = I 

We can establish the following joint beliefs 

Joint Belief of Independent expert #I and #2 estimates in {SR} is 0.995 

Joint Belief of Independent expert #I and #2 estimates in {SR, SE} is 0.9996 

Joint Belief of Independent expert #I and #2 estimates in {SR, SE, SI, SS, FP} is 1 

Since SR is the dominating risk factor within the context of our assumptions, the 

joint belief in SR of 0.995 infers that both CAIG and IDA have similar if not exact beliefs 
on the question at hand regarding SR. 

The next step in the analysis is to determine the “direction” of the belief For the 
purposes of our study, we assume that the DST results imply a 99% degree of belief that 
the Army underestimated the risk of requirements creep, and infer that requirements 
volatility is actually 20%^ based on both the CAIG and IDA’s estimates as opposed to the 
12% volatility value determined by the Army 

Consequently with the belief that an increase in requirements risk would 
adversely impact cost and schedule estimation, we readjust the risk factor off 
requirements volatility by increasing the probabilities of its occurrence by 8% in our 
model. (20% - 12%) and assign readjust the other risk factors of (SE, SS, SI, FP) such 
that there sum is 1. 


4 While our assumption of the risk of requirements increases is consistent with the findings of both the CAIG and 
IDA as reported in their testimony to Congress based on the cost estimates provided by the Army, we further assume in 
this study that the preponderance of evidence gathered by both independent experts served as a basis for determining a 
20% volatility based on the proportionate recomputation of all the other risk factors surrounding the PCS program such 
that the sum is of the probabilities of all the other risk factors is 1 or 100%. 
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We rerun a Monte Carlo Simulation and as shown in our model (Table 22).Our 


new results are depicted in the new output charts; 




RIskFaclois 



NPV of F*f Reims on a 
liwesMof 11(3.7 ion in an 
Asset Valued al 110 lion 

Cuiient 
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|SL0C) (Monihs) Planned Costs 

Reijuleinents 
Deep Risk 
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Risk 
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Risk 
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Unit Cost 
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Table 22. 


Screen capture of risk model developed in the Risk Simulator with revised estimate. 



_y 

Type Two-Tail, Lower -Infinity Upper Infinity, Certainty: 100.0000% 

Figure 50. Risk Simulator output depicting revised returns on a $163.7 billion 
Investment in the FCS Network with revised probability estimates. 
The frequency histogram (Figure 49) shows the frequency counts of 
values occurring in the total number of trials simulated. The vertical 
bars shows the frequency of a particular NPV occurring out of the 
total number of trials, while the cumulative frequency depicted by the 
smooth line shows the total probability of all values at and below the 
maximum NPV occurring in the forecast. 
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Returns on $i63L7 


Jnjxj 


Histogram Statistic | Preferences | Options | 


Statistics 

Result 1 


Number of Trials 

1000 


Mean 

5.751538E+012 


Median 

5.833072E+012 


Standard Deviation 

5.444946E+011 


Variance 

2.964743E+023 


Coefficient of Variation 

0.0947 


Maximum 

6.465936E+012 


Minimum 

2.327804E+012 


Range 

4.138131E-H312 


SkeviTiess 

-1.5697 


Kurtosis 

3.6356 


25*4 Percentile 

5.499394E+012 


75*4 Percentile 

6.156987E+012 


Percentage Emor Precision at 95*4 Confidence 

0.5868*4 



Figure 51. Revised volatility of the returns on the $163.7 billion Investment in 
the FCS Network. The forecast statistics summarizes the distribution 
of the forecast values in terms of the four moments of a distribution. 


In comparing our new volatility estimates of 0.09477 to the previous volatility 
estimate of 0.0866, we observe that the volatility of the returns on the FCS Network 
increased, while the returns of our investment deelined from $6.1 trillion to $5.7 trillion - 
a direet eonsequenee of the inerease in volatility of our investment effort. Therefore given 
the volatility of the returns on the FCS Network investment the decision-maker with 
executive oversight on the FCS Network needs to develop and incorporate the 
appropriate Real Options. We now proceed to formulating our Real Options. 

P. IDENTIFYING OPTIONS 

Now that we have determined the volatility of the returns of the FCS Network, we 
develop Real Options to address the risk factors we have identified so far. The strategy 
taken is that in the light of requirements uneertainty which is beyond the eontrol of the 
program manager or exeeutive, how does s/he deerease eosts or how do they stay within 
the authorized budget. We therefore reeast the FCS Network development effort as a 
series of Deferment/Leaming Options and Investment/Growth Options during whieh the 
Option to Start, Stop, and Options to seale up staff and seale down staff or defer 
development in the face of requirements uneertainty is utilized. This whole strategy is 
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based on reallocating resources based on the periods of lulls in software development 
activities based on uncertainties, i.e. maximizing resources (e.g. staff). This concept is 
based on a stage-gate development approach utilizing the concepts of both iterative 
development approaches and standard portfolio theory. Its success relies on the 
successful and correct partitioning/decomposition of the PCS Network into the 
appropriate subsystems which the Army appears to have done already. We therefore 
partition the PCS Network solution into the six independent systems as identified in 
Table 13 above as follows: 

1) Combat Identification 

2) Battle Command and Mission Execution 

3) Network Management System 

4) Small Unmanned Ground Vehicle 

5) Training Common Components 

6) Systems of Systems Common Operating Environment 

If further partitions of each of these subsystems are warranted, they should be 
developed accordingly. Now that we have our ECS Network “portfolio” of all the 
required systems, to the extent possible, development of each system can begin at the 
same time, using iterative development techniques to guide each system towards success 
while at the same time deferring systems that still face requirements uncertainty. The 
iterative concepts employed would facilitate shorter feedback cycles by cycling through 
the development phases, from gathering requirements to delivering functionality in a 
working release. 

Since there are six possible software systems, we examine the 63 possible 
combinations of one or more components may have uncertainties in them at the same 
time, resulting in a strategy tree based on 63 possible strategies. 

Rather than develop a complicated strategy tree showing our 63 strategic 
pathways, we examine one possible scenario with two different strategic options. 
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1 . 


Scenario 


All six systems are faeing uneertainty with the exeeption of one system. That is, at 
least one system out of the six systems is not faeing requirements uneertainty. We 
therefore develop two types of options in response to this seenario 1) Compound option 
2) Deferment Option 


a. Compound Option 

In the event that at least one of the systems is not faeing requirements 
systems, with all the others faeing requirements uneertainty, an option eould be 
developed to “seale down” the resourees (staff) alloeated to the other systems. The staff 
eould then be switehed to work on the system that is not faeing requirements uneertainty, 
while the uneertainty is addressed using our uneertainty elieitation model. The 
assumption with this approaeh is the system development effort whieh the staff engineers 
are being realloeated to work on is not already behind sehedule and henee does not 
violate Brooks Law^). If the development effort whieh the staff are being assigned to 
work on is late (behind sehedule), the number of staff, experienee level and role whieh 
the added staff would play in the software development effort must be taken into 
eonsiderationWe therefore frame the Real Options in this ease as: 

To illustrate the reasoning behind the development of our Real Options “Option to 
Seale down from a system with uneertain requirements. Option to Switeh resourees to 
another system. Options to Seale up staff assigned to the development of a system not 
faeing uneertainty we use an analogy based on the eoneept of “switehing lanes” on a 
highway. 

When a driver feels a lane is going too slow, he has the option to switeh to the fast 
lane. However this option eomes with a premium in form of faster burn rate of fuel and 
training assoeiated with being able to drive in the fast lane in order to keep up with the 
moving traffle. The driver remains in the fast lane until he runs into another driver ahead 


^ Brooks law states that adding people to a late project makes it later. 
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of him, slowing down the traffic. He then exercises his option to switch back to the slow 
lane. 

In the case of the PCS Network, by relocating resources from unsuccessful 
systems (systems facing significant uncertainty) to “successful” systems, i.e. systems not 
facing requirements uncertainty, the development of the successful systems could be 
accelerated based on the increased level of effort. Granted previous research by Robert 
Glass has shown that this might not be the best way to accelerate delivery (i.e. by 
throwing bodies to project), we explicitly acknowledge this by developing a “Switching 
Option” that acknowledges the costs that might be incurred in the logistics related to 
training and reassigning staff to the successful systems. 

Therefore given our approach in this scenario, what we have essentially done is 
develop a “Sequential Compound Switching Option”, an “Option” on a “Option”. 

b. Deferment Option 

In the event that five of the six systems are facing requirements 
uncertainty, another option could be developed to stop and defer all development until 
uncertainty is resolved in the single system. What we have essentially done is develop a 
Deferment Option. 

Now that we have determined the appropriate Real Options, our next step is to 
develop a strategy tree depicting the strategic pathways. As mentioned in the previous 
chapter under the partitioning (decomposition) of the software solution, the success of 
these strategies hinges on the following. 

The software solution has been partitioned (decomposed) properly, taking into 
consideration uncertainty both within the subsystem and across the subsystems 
and uncertainty either reduced to lowest level possible or resolved using modeling 
and simulation tools as advocated for in our discussion. 

We depict our strategy tree in Figure 52. 
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Q. STRATEGY TREE OF COMPOUND AND DEFERMENT OPTIONS 


Strategy A 
Uncertainty in 


> Phase 3 


Phase 2 


Switch (aiiocate) resources from 


Phase 1 


Continue deveiopment 
► of System of Systems 
Common Operating Environmenl 


1) Combat identification 
i) Battle Command and Mission Execution 

3) Network Management System 

4) Smail Unmanned Ground Vehicle 

5) Training of Common Components 

to 

System of Systems Common Operating 
Environment 


Switch (allocate) resources 
from 

System of Systems Common Operating 
Environment 
to 

1) Combat Identification 

2) Battle Command and Mission Execution 

3) Network Management System 

4) Small Unmanned Ground Vehicle 

5) Training of Common Components 

as 

Uncertainty becomes resolved in 
any of the 5 components 




1) Combat Identification 
Battle Command and Mission Execution 

3) Network Management System 

4) Small Unmanned Ground Vehicle 

5) Training of Common Components 



Exit 


Do Nothing 


Start development 
of FCS ^ 
Network 


Defer 


1) Combat Identification 
Battle Command and Mission Execution 

3) Network Management System 

4) Small Unmanned Ground Vehicle 

5) Training of Common Components 


Defer 



Exit 


Do Nothing 


Strategy B 


Uncertainty in 


1) Combat Identification 
i ) Battle Command and Mission Execution 

3) Network Management System 

4) Small Unmanned Ground Vehicle 

5) Training of Common Components 


1) Combat Identification 
t) Battle Command and Mission Execution 

3) Network Management System 

4) Small Unmanned Ground Vehicle 

5) Training of Common Components 
II) System of Systems Common Operating 

Environment 

until uncertainty is resolved in at least 
_ one components _ 



Figure 52. Strategy tree depicting the types of options for scenario involving 5 
Component Systems of the FCS facing Uncertainty. 
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R. OPTION VALUATION 


As the next step in our methodology, we examine strategy A to value the 
“options” we ereated to hedge against risks using the Binomial Lathee approaeh. To 
aoeomplish this, we used the Super Lattiee Solver 3.0 provided by Real Options 
Valuations Ine. The simulator uses a two step approaeh. 

1. The first step involves the valuation of the underlying asset. 

2. The seeond step involves the ereation of the option valuation lattiee using the 
values eomputed in the lattiee evolution of the underlying asset. 

We seleeted the “Multiple Asset Supper Lattiee Solver” ehoiee in the tools menu 
beeause our strategy approaeh is based on sequential eompound options with multiple 
phases (3 phases based on our strategy tree). 

I. Real Options Assumptions 

Sinee we already made an assumption that the PCS Network developments eosts 
$163.7 billion, and have also determined that the NPV of the PCS Network redueed from 
a initial value of $6.4 trillion to $5.7 trillion due to a volatility of 0.0947%. We apply 
these assumptions in our model based on a risk free rate of 3%. 

a. Strategy A 

Phase 1. Sinee we have uneertainty in 5 of the eomponents, we proeeed to 
develop the remaining eomponent and defer the other 5 eomponents until uneertainty is 
resolved in the 5 eomponents. 

Phase 2. We alloeate the resourees from the 5 components facing 
uncertainty to component 1. 

Phase 3. We reallocate the resources back to the 5 components once 
uncertainty is resolved. We account for the administrative costs associated with 
reallocating resources back to the development of the 5 components. Thus we recoup the 
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overall benefits of the $6 billion set aside in phase 1 for administrative and resource 
allocation logistics issues. 

With this assumption in hand, we go ahead and set up our model as 
depicted below in Figure 53 below subject to the following underlying assumptions: 

Implementation Cost of FCS Network is $163.7 billion 

Value of Underlying Asset is $6.4 trillion 

The risk free rate is the 3.0 

Volatility of our project cv = is 0.0947 

Duration is 13 Years 



Figure 53. Screen Shot of our Model in Super Lattice Solver 3.0. 

We execute the model and obtain the lattice of our underlying asset as well as the 
lattices associated with each of the three phases 
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FCSN Underlying Asset Value Lattice 


6526471 









6513.71 








6500.97 


6500.971 






6488 27 


6488 27 






6475.58 


6475.58 


6475.581 




6462 92 


6462 92 


6462 92 




6450 29 


6450 29 


6450 29 


6450.291 


6437 68 


6437.68 


6437.68 


6437.68 



6426.10 


6425.10 


6426.10 


6426.10 


6425.101 


6412.54 


6412.54 


6412.54 


6412.54 


6412.54 


6400.00 


6400.00 


6400.00 


6400.00 


6400.00 


6400.001 


6387 49 


6387.49 


6387 49 


6387 49 


6387 49 



6375.00 


6375.00 


6375 00 


6375 00 


6375 00 | 


6362.54 


6362.54 


6362.54 


6362.54 




6350 10 


6360 10 


6350 10 


6350.101 




6337 69 


6337 69 


6337 69 






6325.30 


6325.30 


6326.301 






6312 93 


6312 93 








6300.69 


6300.591 








6288 28 



6275.981 


Figure 54. Lattice of Underlying Asset (FCS Network). 

The value of the underlying asset was computed as $6.4 trillion (Figure 54). 


Strategy A Option Valuation Lattice 


6395 93 











6383.34 



6370.78 


6370 . 44 ] 


6358.24 


6357 90 



6345.73 


6345.39 


6345.061 


633324 


6332.90 


6332 56 



6320 77 


632043 


6320.10 


6319.761 


6308 33 


6307 99 


6307 66 


6307 32 



6295.91 


6295 58 


6295 24 


6294.90 


6294.561 


6283 52 


6283 18 


6282 85 


6282 51 


6282 17 


6271.15 


6270.82 


6270 48 


6270.14 


6269 81 


6269.471 


6258.47 


6258.14 


6257 80 


6257.46 


6257.13 




6245 82 


6245.48 


6245.15 


6244.81 


6244.471 




6233 19 


6232 85 


6232.52 


6232 18 






6220.58 


6220.25 


6219.91 


6219 57 | 






6208 00 


6207.66 


6207.33 








6195.44 


6195.11 


6194 771 








6182 91 


6182.57 










6170.40 


6170.061 










6157.91 



6145.45 


Figure 55. Phase 1 Option Valuation Lattice. 

The option analysis which represents the value of the option under strategy A 
returned a value of $6.27 trillion (Figure 55). The lattices are created and values 
computed through backward induction working backwards from phase 3 through phase 2 
to phase 1. Given the option value of $6.27 trillion, the intrin s ic value of the option 
(option premium) is determined to be $6.27 trillion - $5.7 trillion = $570 billion . 
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PHASE2 Option Valuation Lattice 


6420.20 











6407.58 



6394 99 


6394 711 


6382.41 


6382.14 



6369 87 


6369 59 


6369 321 


6357.35 


6357.07 


6366.80 



6344.85 


6344.58 


6344.30 


6344.021 


6332 38 


6332 10 


6331 83 


6331 55 



6319.93 


6319.66 


6319.38 


6319.11 


6318.831 


6307.51 


6307.23 


6306.96 


6306.68 


6306.41 


6295 11 


6294 83 


6294 56 


6294 29 


6294 01 


6293.731 


6282.46 


6282.19 


6281.91 


6281.64 


6281 36 




6269 84 


6269 56 


6269 29 


6269 01 


6268741 




6257 24 


6256 96 


6256 69 


6256.41 






6244.66 


6244.39 


6244.11 


6243.841 






623211 


6231 84 


6231 56 








6219 59 


6219 31 


6219 03 | 








6207 08 


6206 81 










6194 60 


6194 331 










6182 15 



6169-72 


Figure 56. Phase 2 Option Valuation Lattice. 


The phase 2 option analysis returned a value of $6.29 trillion (Figure 56). 


PHASES Option Vafuaoon Lattice 







6522.071 







6609.32 







6496.69 


6496.581 





6483.89 


6483.88 





6471.21 


6471.20 


6471.191 



6458.56 


6458.54 


6468.53 



6445.93 


6445.92 


6445.90 


6446 . 89 ] 


6433.32 


6433.31 


6433.30 


6433.29 



6420.74 


6420.73 


6420.72 


6420.71 


6420 . 70 ] 


6408.19 


6408.18 


6408,17 


6408.16 


6408.14 


6395.66 


6395 65 


6395.64 


6395.63 


6395.61 


6396 . 60 ] 


6383.14 


6383.13 


6383.12 


6383.11 


6383.10 



6370.65 


6370.64 


6370.63 


6370.62 


6370 . 61 ] 


6358.18 


6358.17 


6358.16 


6358.15 



6345.74 


6345.73 


6345.72 


6345 . 71 ] 



6333.32 


6333.31 


6333.30 





6320 93 


6320.91 


6320 . 90 ] 





6308.65 


6308.54 







6296.21 


6296 . 20 ] 







6283.89 









6271 . 59 ] 


Figure 57. Phase 3 Option Valuation Lattice. 


The phase 3 option analysis returned a value of $6.39 trillion (Figure 57). 
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Option Valuation Lattices 
Phases 


Cost 

$6.41 

Dividend 

0.00% 

Riskfree 

3.00% 

Steps 

300 


Terminal Equation Max(Underl/ing-CostO) 

Intermediate Equation Max(Undertying-CostOptlonOpen) 

Intermediate Equation (Blackout) OptionOpen 


Phases 


Cost 

$130.41 

Dividend 

0.00% 

Riskfree 

3.00% 

Steps 

200 


Terminal Equation Max(Phase3-CoslO) 

Intermediate Equation Max(Phase3-CostOptionOpen) 

Intermediate Equation (Blackout) OptionOpen 

Phasel 


Cost 

$27.28 

Dividend 

0.00% 

Riskfree 

3.00% 

Steps 

100 


Terminal Equation 
Intermediate Equation 
Intermediate Equation (&ackout) 


Max(Phase2-CostO) 

Max(Phase2-CostOptionOpen) 

OptionOpen 


Custom Variables 


Name 

Value 

Starting Step 


Contract 

Expansion 

Salvage 

Salvage 

Salvage 

Savings 




0.90 

1.50 

80.00 

90.00 

100.00 

20.00 




0 

0 

0 

11 

31 

0 





Figure 58. Audit Sheet For Strategy A. 


We also captured the “audit” trail of the values in our model which is depicted in 
Figure 58 above. The Terminal equation depicted in our model is the computation that 
occurs at maturity, while the intermediate equation is the computation that occurs at all 
periods leading up to maturity and is computed using backward induction hence the 
reduction in the valuation of the options from $6.39 trillion to $6.29 trillion and to $6.27 
trillion across phases 3, 2 and 1 of the options valuation lattices respectively. 

b. Strategy B 

In strategy B, which calls for deferment, the assumption is made that the 
duration for deferment option would be 3 years. (This means that the project is deferred 
and not commenced until uncertainty is resolved within the 3 year period.) The model is 
set up (Figure 59) using the same assumptions below. 

Implementation Cost of FCS Network is $163.7 billion 

Value of Underlying Asset is $6.4 trillion 
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The risk free rate is the 3.0 


Volatility of our project cv = is 0.0947 
Duration is 3 Years (Maturity of deferment option) 



Figure 59. Real Options Super Lattice Solver Deferment Model. 


We also captured the “audit” trail associated with our model which is 
depicted in Figure 60 below and also provide an explanation of the equations in the 
analysis of our results. 

Option Valuation Audit Sheet 


Assumptions 
Asset Value (SJ 
Implementation Cost ($) 
Ktaturity (Years) 
Risk-free Rate (%) 
Dividends (%) 

Volatility (%) 

Lattice Steps 
Option Type 


$6.400 00 
$163.00 
3 00 
3 00% 
0 . 00 % 
0.94% 
300 

Custom 


Intermediate Computations 
Stepping Time (dt) 

Up Step Size (up) 

Down Step Size (down) 
Risk-neutral Probability 


0 0100 
1.0009 
0 9991 
0 6594 


Results 

Auditing Lattice Result (10 steps) 
Super Lattice Results 


6251.03 
6251 03 


Figure 60. Audit Trail of Option to Defer Model. 
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Strategy B Underlying Asset Value Lattice 


6460-44 











6454-37 



6448-31 


6448-311 


6442-25 


6442-25 



6436 20 


6436 20 


6436 201 


6430-15 


6430-15 


6430-15 



6424-11 


6424-11 


6424-11 


6424-111 


6418-07 


6418 07 


6418-07 


6418 07 



6412-04 


6412 04 


6412-04 


6412-04 


6412-041 


6406-02 


6406-02 


6406 02 


6406 02 


6406 02 


6400 00 


6400 00 


6400 00 


6400 00 


6400 00 


6400-001 


6393-99 


6393-99 


6393 99 


6393-99 


6393 99 




6387 98 


6387-98 


6387-98 


6387 98 


6387 981 




6381 98 


6381-98 


6381 98 


6381 98 






6376 98 


6375-98 


6375 98 


6375-981 






6369 99 


6369 99 


6369 99 








6364-01 


6364-01 


6364-011 








6368 03 


6358 03 










6352-05 


6352 05| 










6346 08 



6340-12 


Figure 61. FCS Network Underlying Asset Lattice of with Deferment Option. 

The model is exeeuted and similar to strategy A, the value of the underlying asset 
was eomputed as $6.4 trillion (Figure 61). The option analysis on the other hand returned 
a value of $6.25 trillion (Figure 62). Thus the intrinsie value of the deferment option 
under strategy B is determined to be $6.25 trillion - $5.7 trillion = $550 billion. 


Strategy B Option Valuation Lamce 







6311-031 







6305 00 







6298-98 


6298-891 





6292 97 


6292 88 





6286 96 


6286 87 


6286 781 



6280 96 


6280 87 


6280 78 



6274 96 


6274 87 


6274 78 


6274 691 


6268 97 


6268 88 


6268 79 


6268 70 



6262 98 


6262 89 


6262 80 


6262 71 


6262621 


6257 00 


6256-91 


6266 82 


6266-73 


6256 65 


6251 03 


6250-94 


6250-86 


6260 76 


6260 67 


6260 681 


6244 97 


6244-88 


6244 79 


6244-70 


6244-61 



6238 92 


6238 83 


6238 74 


6238-65 


6238 56| 


6232 87 


6232 78 


6232 69 


6232-60 



6226-83 


6226-74 


6226-65 


6226-561 



6220 80 


6220 71 


6220 62 





6214 77 


6214-68 


6214-591 





6208 74 


6208-65 







6202-72 


6202 631 







6196 71 









6190 701 


Figure 62. Options Valuation Lattice under Deferment. 
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S. ANALYSIS OF RESULTS 

In comparing the results of both strategies A and B, we observe that while 
strategy A has an option value of $570 billion, strategy B has an option value of $550 
billion. What this implies, is that under both strategies A and B, the software exeeutive 
should be willing to pay no more, (and hopefully mueh less than) the option value of 
$570 billion and $550 billion respeetively in addition to the initial investment eost of 
$163.7 billion to inerease the ehanees of reeeiving the projeeted NPV of $6.27 trillion 
under strategy A and $6.25 trillion under strategy B for the FCSN as opposed to the 
eurrent projeeted NPV of $5.7 trillion in light of the risks eaused by the uneertainties in 
five of the six software eomponents. 

In analyzing both strategies, strategy A is more attraetive in the sense that instead 
of waiting to invest until after 3 years (after whieh uneertainty would hopefully have been 
resolved) and then proeeeding to spend $163.7 billion at onee, the staged phase approaeh 
that adopts spending some little first and then investing more over time with a proof of 
eoneept safer is worth more. Therefore under these eonditions, strategy A whieh employs 
the eompound sequential options is the optimal approaeh. 

T. KEY BENEFITS OF APPROACH 

The key benefit of the approaeh that ean be observed in the analysis of results is 
the reeognition of the added flexibility whieh the software program manager has, to make 
deeisions during the life of the PCS Network effort. The Real Options approaeh viewed 
the PCS Network as a series of sequential eompound options, with eaeh option depending 
on the exereise of those that preeeded it. 

This approaeh is very benefieial to the FCS program in that the eurrent aequisition 
strategy is based on a eost reimbursable strategy in whieh eontraetors are reimbursed for 
their expenses regardless of the sometimes unreasonable justifieation of ineurring the 
eosts. By adopting our approaeh, we posit that the manager or deeision maker would be 
able to aetively manage the investment effort by exereising the applieable Call Options at 
the appropriate time, a key benefit towards eontrolling eost overruns that the eurrent eost 
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reimbursable aequisition strategy faeilitates. In eomparison of our approaeh to traditional 
Diseounted Cash Flow teehniques sueh as Net Present Value, we ean see that the 
investment costs of the FCS Network investment effort is lower under NPV ($163.7 
million) than the Real Options approach ($163.7 million + Option Premium), because the 
NPV approach assumes there is no uncertainty in the FCS Network and thus does not 
account for risk in the decision making process. 

Secondly, we have shown how the volatility of the FCS Network can be refined in 
the event that the risk factors are either underestimated or overestimated by the Army by 
using Dempster-Shafer Theory to refine the Army’s estimate. This is particularly 
important because it serves as a cursory check of the Army’s assessment. By refining the 
risk factors using our approach, the decision maker would be in a better position to 
develop and price the associated Real Options with a higher degree of confidence. 
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VII. SUMMARY 


A. OVERVIEW 

In the Department of Defense (DoD), the typieal outcome of a software 
acquisition program has been massive cost-escalation, slipping planned delivery dates 
and making major cuts in the planned software functionality to guarantee program 
success. To counter this dilemma, the DoD put forth a new weapons acquisition policy in 
2003 based on an evolutionary acquisition approach to foster increased efficiency while 
building flexibility in the acquisition process. However, the evolutionary acquisition 
approach relies on the spiral development process, which assumes the end-state 
requirements are known at the inception of the development process, which is a 
misrepresentation of reality in the acquisition of DoD software-intensive weapons 
systems. Hence the need for a framework as proposed in our study at a higher level of 
abstraction (acquisition decision-making level) aimed at mitigating risks associated with 
the technology objectives, constraints, and alternatives as put forth by the customer. 

Our approach addresses these issues by taking a proactive/preemptive approach to 
risk management by planning and paying for risks up front. This is not to say that risk 
management strategies are not being adopted today, but rather a failure of management to 
take a strategic approach towards risk management. The status quo today is to employ 
reactive risk management strategies that often result in the reduction of much needed 
functionality from the scope of the software investment effort. Therefore we proposed a 
more proactive decision making-framework to guide management in the form of our Real 
Options framework. 

The Real Options approach is based on the concepts of financial options theory, 
and in our study we have shown how it could be used as proactive risk management tool 
that could be employed at the strategic decision-making level (executive level) pre¬ 
acquisition, further complementing the spiral development approach at the “tactical 
level”. The Real Options approach builds on several tried and proven approaches of 

management. In this study we have shown using the U.S. Army Future Combat Systems 
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program as an example that the traditional Real Options methodology, when enhaneed 
and properly formulated around a proposed or existing software-investment, eould 
provide a framework for guiding software aequisition deeision-making by highlighting 
the strategie importanee of managerial flexibility. This flexibility offers management the 
ability to balanee the satisfaetion of a eustomer’s requirements within the realms of the 
assoeiated eost and sehedule eonstraints thus validating our hypothesis by developing the 
appropriated options during the aequisition deeision making phase and exeeuting the 
options when it beeomes optimal to do so. 

B. FINANCIAL MODEL AND ASSET VALUATION 

To validate our hypothesis, we developed eorrelations and establish eomplianee 
between the established Real Options pre-eonditions and the software investment 
deeision-making proeess. While we were able to establish eomplianee with the Real 
Options pre-eonditions, this was not an easy task, mostly due to laek of detailed data — 
when data was available, it was not available at the level of granularity whieh we would 
have desired. Consequently, we had to make some assumptions regarding the data in 
order to validate our hypothesis. However, we believe that our hypothesis would still 
hold true when employed with real “data”. 

The main Real Options pre-eondition whieh we had to make assumptions about 
was the preeondition of the existenee of a basie finaneial model. While we were able to 
ereate a finaneial model based on our beliefs and intuitions, the laek of a eomprehensive 
methodology for valuing the returns of a potential DoD software investment proved 
problematie in valuing the software asset. Further eompounding the problem was a laek 
of the elear delineation between software and hardware from both a eost and realized 
benefits perspeetive. 

C. UNCERTAINTY IDENTIFICATION AND RISK QUANTIFICATION 

The seeond most import pre-eondition of Real Options, is the existenee of 
uneertainties. Uneertainties must be present, otherwise the Real Options approaeh 
beeomes useless. We foeus on this issue in our study beeause uneertainties have the 
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consequence of introducing risks to the software investment effort, and the risk must be 
properly quantified, because it is a key factor in determining the price of the options we 
would need to create to hedge against the risk. Thus by properly quantifying risk, we are 
increasing our degree of confidence in the volatility of our software investment effort and 
consequently the price of the Option being developed. To satisfy this precondition, we 
proposed the Uncertainty Elicitation Model based on our “2T” approach explicitly aimed 
at capturing uncertainty. This model introduces an explicit, uncertainty elicitation phase 
after the requirements elicitation phase and captures uncertainties along both managerial 
and technical dimensions. Once these uncertainties are captured, we quantify the 
uncertainties as risks by developing subjective probability estimates and modeling them 
stochastically using a Monte Carlo simulation, in order to determine the overall volatility 
of the returns of the software investment effort. 

Next we proposed the use of the Dempster-Shaffer Theory of Evidence (DST) as 
a volatility refinement technique to further refine our initial volatility estimates due to its 
ability to address epistemic uncertainties and reflect ignorance in the risk probability 
estimates. To satisfy the constraint of the independence of the sources of information 
imposed by DST, we utilized the data provided by two independent sources, the Cost 
Analysis Improvement Group (CAIG) and the Institute for Defense Analysis (IDA), to 
improve our initial volatility estimates. We formulated the appropriate Real Options in 
response to the risks using the revised volatility estimates obtained using the DST 
methodology thereby satisfying precondition 4, which calls for the ability of management 
to have the flexibility or option to make mid course corrections when actively managing 
the project. The Real Options are presented in a strategy tree, to highlight all the possible 
strategic pathways for that could be employed to address the risk at hand. 

Given the uncertainties and risks we identified in the ECS program using our 
uncertainty elicitation model, we proposed two “call” options based on a scenario which 
assumed that of the six ECS component systems, one is not facing uncertainty while the 
remaining five software components were facing uncertainty. The proposed “call” 
options allowed the software executive to either “defer” the development of all the 
components of the ECS Network to include the single system that is not facing 
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uncertainty or employ a deferment and switehing approaeh, by deferring development of 
the system faeing uneertainty until uneertainty is resolve and at the same time eontinuing 
the development of the remaining five components. 

D. OPTION PRICING 

To price the Real Options created to hedge against risk, we modeled the options 
reflected in our strategy tree in a simulator ealled the Super Lattiee Solver 3.0 provided 
by Real Options Valuation Inc. 

We input all the neeessary parameters to inelude the key parameters of volatility 
and asset value whieh we had computed earlier on during our analysis and eleeted to use 
the binomial lattice approaeh to value our option. It must be noted that while the Black 
and Scholes method is probably the most popular options pricing approach, we ehoose 
the binomial lattice approach because of its ability to account for American style options, 
whieh is an advantage over the Black and Scholes approach. It is also important to 
emphasize that while there are several types of options, labeled after geographic regions 
(e.g., American, European and Bermudian options). The name of these options are just 
labels for the options and do not imply geographieal restrietions. An Ameriean style 
option ean be used in Europe, just as a European style option eould be used in the United 
States. The names associated with eaeh of these options only have to do with when they 
may be exereised by the option holder. 

Associated with the task of pricing the options is also the timing of the execution 
of the option, with the goal being to find the optimal revenue drivers, which is where we 
further make the assumption, that under the laws of rational judgment, management 
would time and exeeute the options when it is optimal to do so thereby satisfying 
precondition 5, which addresses the issue of managerial judgment from a eompetenee 
perspeetive requiring that management must be smart enough to execute the Real Options 
when it beeomes optimal to do so. 
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E. CONCLUSION 


Uncertainties associated with software-related capital investments lead to 
unnecessary and sometime preventable risks. Since DoD often sets optimistic 
requirements for weapon programs that require new and unproven teehnologies, the 
application of the Real Options methodology would be beneficial as it would enable DoD 
to incorporate the appropriate “Options” into the acquisition contracts. Barring the use of 
an explicit uncertainty elicitation phase as proposed in our research and the development 
of “Options” to hedge against the risk as they appear, we believe the current acquisition 
process would continue to be plagued by the risks of cost and schedule overrun. By 
employing our proposed approach, DoD would be able to optimize the value of their 
strategic investment decisions by evaluating several deeision paths under eertain 
conditions to lead to the optimal investment strategy. 

As it stands today, all the proposals put forth by the Congressional Budget Office 
(CBO) call for the reduction of the overall scope of the PCS program, and while we 
cannot affirm or disprove their recommendations due to the limited data we had at the 
time of our study, we believe that the DoD can benefit from exploring our approaeh by 
utilizing preeise data with our framework to examine the eredibility of the 
recommendations put forth by CBO. Having validated our proposed framework (subjeet 
to the assumptions we made), we believe that if the Real Options methodology had been 
applied to the PCS program from the onset by acquiring the right to exeeute switehing 
options (within the constraints of our assumptions) to hedge against risks, the PCS 
program would most likely not be facing the current calls for the reduction of the overall 
scope of the PCS program due to cost escalation resulting from the inadequate upfront 
risk management planning prior to the investment decision. Hence we believe our 
framework could serve as basis for future work in terms of the expansion of its 
capabilities to address risk and decision-making within the software engineering domain. 

F. FUTURE WORK 

As part of the future work in conneetion with this researeh, we would like to 
formalize and create an automated software acquisition decision-making tool explicitly 
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aimed at managing the risks assoeiated with software-related eapital investments using 
out Real Options approaeh. Speoifieally, we would like to gather historieal information 
on previously eompleted software aequisition programs depleting the number of 
requirements planned at the onset of the aequisition effort and the number of 
requirements delivered at the end of the software aequisition effort, as well as the 
assoeiated eost and sehedule information for eaeh of the aequisition programs. We would 
use all of this data to establish a repository of historieal programs whieh would serve as a 
basis of eomparison with eurrent/future aequisition programs to help provide some 
insight into the issue of requirements volatility and its assoeiated impaet on eost and 
sehedule overruns. By gathering historieal information into a eentralized repository, we 
hope to alleviate the assumptions we made in our study due to the data gathering 
problems we eneountered in this study. We would ineorporate the DST volatility 
refinement teehnique into our software tool and link our automated software aequisition 
deeision making tool to the repository eontaining historieal data of previously eompleted 
software aequisition programs to provide a one “stop-shop” modeling toolkit to better 
faeilitate the aequisition deeision making proeess. 
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